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I. INTRODUCTION 



The Computer-Assisted Instruction Center at Florida State University 
been pursuing a broad range of reseai’ch topics derived from the appli 
cation of computers and related media technology to instructional processes. 

This general research focus has led to a major study of the developmental 
requirements and benefits from an autonomous course in undergraduate 
physics. Within this CAI physics project, we have been studying the nature of 
learning problems, feciUtating teaching strategies, and the role of media for 
the fostering of optimal end-of-course performance. In a related vein, we 
have been utilizing the x>ystems model and CAI in the deriving of newer forms 
of evaluative techniques that are operationally beneficial to curriculum authors. 
In pursuing these broad goals dealing with instructional and evaluative techni(jies 
based on CAI, we have performed a series of learning and i>ersonality related 
experimental studies. Most of these short-term experimental studies are 
originated in and coordinated with a graduate training program in educational 

technology. Thus, the CAI Center has been striving to coordinate and manage 

■ 1 ’ 

these multiple research and developmental activities in order to contribute to a 
better understanding of the role of computers and media technology within a v^ide 
variety of instructional systems. 

This Semiannual Progress Report has been organized into three major 
sections. The first section represents the plans and phased objectives for the 
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maior projects within our Center. In addition to continuing plans for a 
collegiate physios course, the Intermediate Science Curriculum Study course, 
and the CAI course in social’ welfare , we have inaugurated activities relating 
to teacher recruitment and placement, the use of systems concepts within 
education, and accompanying data management and anal3^sis programs required 
to facilitate our research efforts. As can he viewed in the Table of Contents, 

there is a substantive report on each of these projects. 



As a representative sample of some of the more delimited experimental 
studies, the second section presents three studies which are.-nbw in progress. 
These range from the use of CAT in collegiate counseUng processes to the use of 
the CAI systeW to siudy discriminatibli processes under (varyii^ attitudinal/ 

reinforcement conditions. ’ - . * ^ 

‘ ■ The majdr bulk of this ^miahnual Progress Report is a series ;of 

research reports representing'our empirical findings during the past academic 
year. These reports range from reSiiltS derived from the first field testing of 
the coilege physics , multi-media CAT course thrOdfeh a series of specific 



experiinehtkr studies focuSing on learning and adaptive variables open to 
manipulation within CAI . Each of these reports is self-contained and' can be 
review'ed in the Table of Contents. In essence , these reports represent the 



substantive research progress achieved by the CAI Center. h 







n. ]\IAJCR CAI PROJECTS— PLANS 
AND DEVELOPMENTS 

A. Computer-Assisted Instruction in Introductory Physi^ 

The first field test of the computer-based introductory physics course 
was completed in December of 1967. A description of the preparation of this 
first version of the CAI course as well as the field test learning results can be 
found in Section IV. This progress report reviews in detail all of the learmng 
results as well as consideration of operational logistics and attitude reactions on 
the part of the students. In regards to future plans for the project, the staff has 
been pursuing appropriate course revisions based on the item performance and 

error analysis results from this first field test. 

Authors have been analyzing specific learning difficulties in order to 
improve the program. All of the concept film presentations have been 
reanalyzed and more appropriate accompanying audio and printed materials are 
being prepared. In addition, the breadth and depth of the audio lectures are 

• ^ " 1 I! * , ’ T 

being both extended and elaborated. A wider range of questions deaUng with 
concept mastery are also being included in order to allow for even greater 

individualization . 

In the spring of the 1967-68 academic year, we plan to provide a more 
rigorous evaluation of a second version of this multi-media computer-based 
physics course. We will carefully select 40 participants. These will be selected 
according to sex (i.e. , 20 males and 20 females) and prior physics instruction in 
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high school. For cross -group comparison,, a comparable selection procedure 
will be followed for the partial CAl group which receives review and practice 
prior to the mid-term and final exams , and a conventional group that will have 
no exposure to CAI. These groups will be matched according to verbal arid" ' 
quantitative scores on the Florida Twelfth Grade Entrance Examination, as 
well as age. 

In addition, we plan to investigate lesson content by media by 
aptitude interactions. In order to facilitate this overall analysis, we plan to 

; “ ” I . 

measure all participants on (1) a quantitative reasoning test; (2) a listening 

• • “ . .'U* . 

test; (3) a reading test, and (4) a problem-solving test. In regards to the affect 



domain, we plan to embed anxiety measures within each lesson in order to gain 

l 

an anxiety/sta us report on each student. In addition, we will collect measures 
on impulsivity in order to detect whether learning rates are determined from 



this personality factor. The attitude measures towards this technological 



approach to instruction have been both extended in regards to man-machine 
characteristics. We trust that the spring field te?t will provide us with definitive 
results by which to confirm hypotheses related to the appropriateness of 



computer -controlled media types for students with given learning and personality 
characteristics . 



' 1 1 j * n 
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The Intermediate Seieace Curriculum Study— Future Plaim^ 

During 1968-69, the Intermediate Science Curriculum Study \k’ill 
continue to use the CAI system as a vehicle for evaluating the science materials 
v;hich are being designed for regular classroom usage. (See papers by Snyder 
and Dick in Section IV of this report. ) The major purpose of the current CAI 
instruction of seventh grade students is to gather specific response data in order 
to provide the course authors with empirical data on the students* learmng 
performance. The detailed computerized data analysis system which is required 
for this task has been under development during the current academic year. 
Summarized data from this system is currently being provided to the ISCS staff 

for their use in the preparation for tbe revision process. 

In addition to continuiiig the CAI evaluation of the revised seventh grade 
curriculum, current plans are to have a select, sample of eighth grade students 
who will aiso receive their instruction via CAI during 1968-69. The purpose and 
procedures for instructing this new group of students will be similar to those of 
the seventh grade: to gather response data for revismg the instructional 
materials. Plans have not yet been made for the evaluation of ninth grade 
materials using CAI; however, it is possible that. such an evaluation may be 
conducted beginning in the fair of 1969. ■ , . 

Planning is also underway to embe4 carefu;jy designed test items into 
the seventh grade' CAI instruction in the coming year. These test items will be 
based upon the behavioral objectives which are being established by the course 
authors. Each objective will be evaluated either through CAI -administered 
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verbal items Dr, on occasion,, through the manipulatton of laboratory equipment. 
This criterion performance data will provide additional data for the succeeding 

revisions of the seventh grade materials. . 



C bAI Behavioral Science Course For- Soci a l V/elfare Students 

Recent advances in the field -of media and computer-assisted instruction 
have opened opportunities for professional education never before available. 

This new technical approach has’ particular significance for the sopial and 
behavioral sciences since resdarca in these areas is resulting in^quantitier of 
data and facts that could be more effectively disseminated, mastered, apd 
utilized by students via computer-assisted instruction. - 

In the spring of 1966, preliminary work was begun with gradpate 
students in social work to determine the feasibility of programming behavioral 
science knowledge considered prerequisite in advanced social work education. 
The findings in a pilot prototype study in which 15 graduate students were given 
four different programs provided a positive indication of the feasibUity of this 
approach. For the feasibiUty study, CAI programs were written.in sociology, 
psychology, cliild development, and Freudian analytic concepts. Fpulty 
members of appropriate departments in the University (Psychology, Sociology, 
Home and Family Life) were asked to review these programs. The changes, or 
corrections, and new directions that were recommended were incorporated mto 
the progtams that were used in the pilot prototype. 
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In Septembor, 1967, the four CAI programs were further tested on 113 
students who entered the gradiate school of social work. The results of this 
further evaluative effort have encouraged the researchers to expand the effort 
into media simulation games and professional decision-making tasks. At this 
point, only the feasibility of developing curricula in the indicated areas of 
behavioral science has been established. The pilot prototype programs have not 
been tested for effectiveness with control groups. Further developmental 
research on these programs will help us produce a CAI behavioral science 
curriculum that may be generally useful to social work and other mental health 

disciplines. 

The general goals of the proposed project are: To develop curricula in 
the human behavior and social environment sequence of social work, utilizing 
behavioral science knowledge to design a series of computer-based simulations 
to teach problem-solving and decision-making strategies. More specifically, 
the following objectives for curriculum development are to be met: 

1. To construct a series of computer simulations using social work and 
mental health clinic situations. 

2. To w'rite a series of computer programs integrating computer 
programmed learning with video tape, continuous loop films, single 
concept films , or other visual aids. 

3. To construct and test a series of computer simulations of clinical 
examinations, situations, and models to determine the usefulness of 
such simulations for students entering field placement in mental health 
clinics and hospitals. 

4. To evaluate the effectiveness of the curricula designed by randomly 
assigning students to an experimental group using CAI and a control 







group not using CAI, Appropriate research techniques would be 
employed to evaluate the placebo effect. 



This project is quite ambitious in nature, given the. requirement for 
social welfare situational simulaticn programs. We consider the need to 
evaluate and document tlio value and limits of computer-simulated professional 
decision-maldng and prcKcm- scivinr to be Ivgli ir. the priorities of needed CAI 
research. Given the range of ucademio fcachgrounds and i.idmdual differences 
in academic proficiencies cf tlie soci-al weiiate graduate students, we anticipate 
that a hierarchically organized sot of simulation tas.es will add s^nificantlj to 
the terminal competencies of the students. 



D. Computer-Augmented Teacher Recruitment and-P lacement ~ 

All agencies of tlie educational enterprise have committed themselves 
to the fostering of excellence in the educational process via enlightened teacher 
recruitment and placement services. This is a proposed extension of the. 



computer-based teacher placement project developed by Dr. William L. Maloy 
and James R. Swanson. (See the report in the results section.) The conceptions 
and plans for further developing this w’orl: form the substance of this report. 

Three primary dimensionr. of technological approaches to improved 
teacher selection and plncomont can be clearly identified. The first of these 
relates to the breadth and t^^pe of communication. In essence, an office of 
teacher recruitment and placement not only plays a broker’s role of bringing 
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teacher candidates and local school district administrators together but, more 
importantly, pro\’iding the prerequisite and subsequent information by which 
these contacts can be maximized into effective job placements. The 
utilization of informational processing techniques should focus on the topic of 
providing relevant and personalized information in order to achieve the goal of 
appropriate teacher placement. 

The second major dimension of needed development relates to the range 
and timing of the informational services. Via the use of informational processing 
techniques , one can anticipate a gradation of informational services that ranges 
from hourly requests to satisfy emergency teacher openings to semiannual 
requests to cover anticipated teacher positions. Perhaps moic, importantly, the 
nature of this informational service should be hirhly flexible in order to 
accommodate both the ease with which information about new positions coinSs 
into the general system and the appropriateness with which it is supplied to the 
candidate. The use of informational processing techniques is absolu^ly required 

» n « - 

in terms of the growing volume that currently has been identified. 

The third major dimension of developmental need relates to the 
coordination of various educational agencies. A major first step would be 
coordination of our teacher -training institutions within the State of Florida in 
order to facilitate and foster the placement of highly qualified professionals into 
the local schools of the State. This coordination requirement will lead to an 
operational solution of the problem of compatibility of roles and information by 
the various school districts and agencies. 
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This communication should eventuate in the early identification and 
initiation of a dialogue between future teacher candidates and the potential 
teaching roles in the State of Florida. Thus, one objective is to develop the 
general information system in such a way as to provide appropriate, easy , and 

continuing contact with future teacher candidates. 

Given the initiation of contacts from future as well as currently 
available teaching candidates, the general information system should be capable 
of providing appropriate teaching positions complete with accompanying infor 
matiqn. As placements are made, appropriate reassignment of the candidate 
information should be accomvtished in order to provide for eventual longitudmal 
studies regarding teacher effectiveness and stability. 

The effectiveness of an improved communication process should 

• ' * i 

eventuate in more appropriate matches between teacher candidates and the 
demographic and educational climate characteristics of given school situations. 
The iptent of the informational system is to improve the decision-making 
process, both on the part of the teacher candidate as well as that of the school 
system. Consequently, a major objective will be the development of appropriate 
and relevant information to improve this decision-making process in teacher 

selection. 

In order to effect this better communication, the informational system 
will have to depend on third-generation computer equipment that allows for real- 
time service. real-time service, we explicitly mean the availability of 
near-instantaneous inquiries in order to satisfy emergency and unusual demands. 
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In addition, an appropriate gradation of inquiry and functions over time intervals 
that span all the way to projection studies appropriate for semiannual estimates 
of job openings. Thus, one objective will be to develop the iirfdrmation system 
so that queries can be made as to the availability of teaching candidates for 
specific job opportunities with appropriate characteristics. These requests can 
be made on a moment-by-moment basis. Secoi-dly, a monthly service -by which 
future candidates, currently available candidates, and school district personnel 
can be more adequately informed as to the developing job openings and human 
resource pools will be developed. Lastly, a corollary of this objective is the 
development of projection studies, with sufficient lead time, in order to estimate 
the availability of personnel to initiate the educational process in the fall and to 
continue it during the middle of the year. Thus, the speed of the communication 
process will be vastly improved. 

As indicated above, the coordination of the efforts of teacher-training 
institutions and local school districts in effecting more adequate teacher place- 
ment is one of the major objectives of this project. This will require the 
investigation of mechanisms by which compatible and relevant information can 
both be received and disseminated to these various institutions. Over and abpve 
this , there is a need by wMch feedback can be supplied to the teacher-education 

* * i * 

* ft 

institutions in order to more effectively adapt their programs with sufficient 
time, so as to meet the problems of education today and within the next few 
years. In this regard, a related objective v;ill be the development of high utility 
information that would supply appropriate information to both the teacher-training 
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institutions as ivoll as local school districts as to such outcomes as the 
perf or mancfr characteristics and. ratings of teachers placed in local school 
distrif'ts. Secondly, studies will be performed as to the process by which 
teachers and local school districts go through the choice-making process. 

Moreover , school districts will want to analyze teacher mobility in order to 
more- effectively stabilize their teaching population.. In this regard, information 
dealing with candidates’ qualifications, over and above that required for 
certification, will provide a useful source of information to school districts in 
regard to necessary in-service train: _ig. In turn, the teacher-training institutions 
will have a source of information for estimating future offerings. Thus, there is 
a whole host of evaluative studies which should eventuate out of an information 
system, specifically directed towards teacher recruitment and placement. 

In summary, the following objectives for this p?.*oject have been 
identified: 

1. To develop the general information system in such a way as to provide 
appropriate, easy, and continuing contact with future teacher candidates. 

2. To provide appropriate continuing communication between teaching 
candidates, the State Department of Education, and local school districts. 

3. To develop appropriate and relevant information to improve this 
decision-making process in teacher selection. 

4 . To develop the information system so that queries can be made on 
(a) emergency moment’s notice basis, (b) a monthly summary, and 
(c) a projection basis for predicting future personnel needs. 

5. To expand the range and depth of information in order to improve the 
effectiveness of the recruitment process. 
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6. To study the utilization of this information to coordinate State agencies 
and institutions as well as to provide information to improve the 
in-service training and follow-up services for teachers. 



This project will be initiated during the coming year. The techniques 
of project planning and management (PERT, etc. ) will be employed. 



E. Understanding and Utilizing the Conce pts of ”A Systems Approach»_to 
. . Education Via Computer-Assisted Instruction: A Prereq uisite to 
Implementing a Systems Model for In-Se rvice Education 

The "systems approach" to education has recently been proposed as an 
appropriate theoretical and methodological solution to many of the current 
challenges and problems in Florida education. While understanding that the 
"systems approach" has grown up within the context of the development of higlily 
complex weaponry systems for the Department of Defense and has been utilized 
in the mMiagement of industrial and business concerns , its proposed application 
in education has led to boBi confusion and an array of negative reactions. In 
essence, the educator in many oases fails to see how systems concepts relate to 

his understanding of curriculum and instruction in a school setting or to see how 

. ' * 

systems analysis can facilitate supervision and improvement of the educational 
process. We propose to develop a set of learning materials which will allow 
administrators and leaders of education to better understand the nature and 
conception of systems analysis and modeling as they have applicability in the 




educational domain. 
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The immediacy of the need for understanding the "systems approach" 
to education is reflected in the Litton Report on a Systems Model for In-Service 

* * ' ' i “ 

Education. For the administrator and supervisor in Florida , the suggested use 
of PERT, task analysis, and functional flow diagrams requires both a clear 
understanding of these conceptual tools as well as a degree of sophistication as 
to their potential utility. It is the contention of this proposal that prerequisite 
understanding of these concepts and related techniques in systems analysis will 
be required before the Litton -Report can be implemented. Thus , we propose to 
develop a multi-purpose^mputer -assisted instruction course which would allow 
for a better understanding of how. these systems management techniques can be 

usefully applied in Florida education. 

The Litton Report and Other related documents clearly demonstrate that 
educators in Florida are concerned about both methods and media applics-tions , 
but have little personal acquaintanceship with some of their newer forms. The 
Litton Report indicates that less than three percent of the teachers in Florida 
have had any personal acquaintanceship with computer -ass is ted instruction. 
Moreover, the problem of individualization of instruction is seen only in terms 
of rdte of progfess, and rarely in terms of appropriate selection of content 
depending on individual levels of performance. Thus, we propose to present the 
course* on systems concepts via computer-assisted instruction (CAI) so that each 
of the participant educators can gain personal e^q)erience about this new form of 
media and judge its relative merit as it might relate to the larger cpiestions of 
appropriate 'media selection for the goals of education. 
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In the final analysis, the usage of systems concepts will ultimately 
come from their application in an educational, problematic context. It has been 
found that the use of simulated problems provides for operational experience 
with none of the consequent costs of mistaken judgments or lack of familiarity 
with the technique. We plan to utilize examples from problems dealing with 
media purchases and applications in Florida schools in order to teach analysis 
and problem-solving techniques to our participant educators. Thus, the CAI 
program will offer operational experience and competency to those who wish to 

use it. • . ■ : 

Description of the Course . This CAI program will clearly indicate to 

the participating educator the flexibility and individual selectivity of CAI. 
Focusing on the topics of systems analysis, apd systems podels , three levels of 
inquiry are proposed. The first can be thought of as a general introduction to 
:and review of the major conceptions in the systems approach to education. 

Given a person who wishes to only be acquainted with the major conceptions and 
techniques Leyel One offers a good systematic presentation ^d review. 

Knowing that learners relate best to a meaning&l context , we would 
propose a second level which would present many operational examples and 
• specific utilizations of these concepts in the areas of elementary education, 
secondary education, and administration. Thus, a person wishing greater 
insight could pursue, any given topic in Level Two within his own appropriate 
professional area. In this manner , the participant educator will have a chance 
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to see how the systems management concepts relate to problems with which he 
is immediately acquainted. 

The third level of inquiry relates to operational problem-solving. We 
propose to implement a series of simulated tasks that allows for the direct 
application of the systems concepts. Thus, for that educator who wants to 
become an actual practitioner of the "systems approach," he will have a chance 
to exercise his new found knowledge in a series of tasks that are hopefully 
completely analogous to current problems in Florida education. For example , 
an administrator might be required to analyze the purchasing and in-service 
training problem for a newer media device such as a concept film. Thus , Level 
Three offers a cliance to consolidate the learning and apply the systems 
techniques. By utilizing CAI, each educator will be able to pursue the range of 
topics to his own self-determined depth. Thus , CAI allows for flexibility in 
course objectives and the opportunity to experience an instructional system while 

learning about instructional systems. 

Objectives . In essence, this project will develop a CAI course that 

. I »* » 2 " 

individualizes the approach to systems analysis and systems models in education. 
Specific objectives are as follows. 

First, the essential concept of systems analysis and systems models to 
be acquired can be divided into the following eight ^ubconceptual areas: 

1. Management planning (analysis of management goals, manpower require- 
mejltg and PERT charting) 

2. Functional analysis (determination of context constraints, feasibility, 
and functional flow diagramming) 
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3. Behavioral goals (determination of levels of behavioral performance 
and relations to task performance) 

4. instructional strategies (determination of appropriate instructional 

learning pathways to accomplish the projected goa s) 

« ’ . * « 

5. Media selection (selection of appropriate instructional media m order 
to implement the instructional strategies) 

6. Activity analysis (consideration of the overaU activity in order to 
determine operational procedures) 

7. Systems specification (determination of appropriate manpower and 
equipment requirements) 

8. Evaluation and final design synthesis (determination of revision cycle 
and potential growth for an educational project). 



The primary objective will be to effectively teach- these concepts,. 

Each of these concepts will be prepared in order to allow for three 
levels of inquiry, namely: (a) a general understanding; (b) specific n ationships 
to a si>ecialized area of education; and (o) operational problem-solving. There 
will be a heavy stress on the relationship of the "systems approach" to ultimate 

development of new programs for in-service education. 

Ah additional objective will be the determination of the effectiveness of 
this CAI approach to teaching systems concepts as well as its concerrent effect 
on positive and negative attitudes towards technological techniques of instruction. 






18 



F, HAT Systems Prog ramming 

•The delivery of a new CAI hardware and software system, like the IBM 
1500 system, recjuires considerable extension of the manufactnrer's programming 
package in order to accomplish research goals as well as to provide an efficient, 
flexible operation. As described in our July. 1967 Semiannual Report, we 
contend that a generaUzed data management system is required for even 



preliminar^r data analysis. 

. The major goal^ of our developing data management system (which is a 
series of flexible, interlinked data i^knipulation and statistical analyses 



programs) is .to provide the researcher with his specified data analyses reports 
that will foster :extensions. course revisions, and hypothesis testing derived from 
Ms CAI program-. In essence, the data management system should provide 
complete internal manipulation of data (i.e. . no required output/input of data 
such as by cards) wMle efficienfly handUng the logistics of multiple research 
endeavors and the open-ended growtli of the response history file structure. 

Instead of reviewing the architecture of our data management system, 
we include 51 concise description of current computer programs. The -tatus of 
: each-pnogram is either (1) completed (i.e. . operative without any known 
errors). (2) testing (i.e. . coded but requiring more error checking), or 
(3) designed (i.e. . planned and specified via flow diagrams and fonctional 



requirements) . 

In the section under data management, the programs relaUng to the 
condensing, editing, and manipulation of the learners' Mstory files are 
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presented. The section on CAI computer programs and special functions 
enumerates the programs that provide for extensions of the CCURSEWRITER II 
operating system. The third section on utility programs presents the necessary 
input/output routines for efficient use of the computing equipment. The fourth 
section presents a list of miscellaneous programs founa necessary in order to 
further our development of both the CAI operating system as well as the data 

management system. 

A. DATA jVIANAGSMENT : These data analyses preparation programs 
convert CAI response records produced by students into a 
workable file structure. 

Conversion and Edit . —This program converts variable length 
= . ; raw records to fixed length history file records. (Status; 
completed) 

2, Merge . —This program merges sorted updated CAI response 
tapes to history file master reels, (Status: completed) 

3, Sort. -^This program sorts condensed CAI responses by 
course name by student. (Status; conqpleted) 

4, Delete/Select . — ^This program deletes or selects specified CAI 
responses from master history file. (Status; testing) 

5, I/R Monitor . —This monitor system controls and schedules 
all of the data management programs. (Status: completed) 
Curpa . — ^This program pulls specified CAI responses from 
history file and builds an output tape, (Status; completed) 
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7. 1130 Statistical Package , —This program package (multi- 

variate analyses, regression models, and correlation 
techniques) is adapted to the 1800 system. Statistical 
programs are to operate on master response file data. 
(Status: completed) 

B. CM CONTROL PROGRAMS AND SPSCLA.L FUNCTIONS: These 



programs are written to be used in CAI mode. 

'■ r. Location .— This command is used in proctor mode to locate 
■ specific labels and to gi^/e the^r relative disk addresses. 

m • 

(Status: completed) 

■ 2. ReI6ct . --This background application is used to move a CAI 
* ' ' ’ ■ file from one* tia'ck to another. It also updates all linkages. 

(Status: completed) 

■■ 3^ ' Help' . ' — TIfis command is used iri student mode. It will allow a 

■ ' student to go from one label to another. (Status: testing) 



0 } UTILrflES 3PROGRAMS : The§e are general utility programs that 
* alibw for more efficient system operation and data management. 






I.' Tape to cards 



^ape to printer 

I. -r cards and printer ' 

4. Tape to tape 
'■ ' 5. Tape to disk 
’’ ’ fe.' * cards' to tape 
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7. Disk to cards 

8. Disk to printer 

9. Disk to cards and printer 

10. Disk to disk 

11. Cards to disk 

12. Disk to tape 

13. 80/80 reproduce 

14. 80/80 list. 

(Status: completed) 

D Tjngr^T.TATjynnS PROGRAMS : Below is a list of feese programs 
and what they do: 

1, MA.GTI. — This program allows building and manipulation of 
unlimited array sizes in FCRTEAN and Assembler langu^e. 
(Status: completed) 

Z MAGT, — ^This program allows end of reel to be recognized. 

TMs is a modification of IBM-supplied I/O routine. (Status: 
completed) 

3 3BIN, This program converts EBCDIC to binary. (Status. 

completed) 

4, bine, —This program converts binary to EBCDIC. (Status. 

completed) 

5, LCAL, —This program calls programs from TSX pack by 

program number and loads it into core. (Stetust completed) 
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B TAP’S TEST. — ^This program reads tape and damps records 

with errors. (Status: completed) 

7 SPLIT TAPE. — ^This program splits an input reel into two 

output reels. (Status: completed) 

8 PRHSX. — ^This program prints tape in hex. (Status: completed) 

9. SSQ. — ^This program punches sequence numbers in cards 

(cc. 73-80). (Status: completed) 

10 MOVE. — This program moves one area of core to another. 

(Status: completed) 

11 ydMTP. — ^This program dumps variable length tape records. 

1 . m ^ • » 

(Status: completed) 

12 ABSTRACT. --This program aids in program documentation. 
... (Sta^: designing) . 
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III. PLASHED STUDIES OF COMPUTER APPLICATIONS IN 
INSTRUCTIONAL A® ADMINISTRATIVE PROCESSES 

A. A Comparison of the Effects of Computer-Assisted Instruction and 
Lecture-Discussion Method s of Presentation in a General Curriculum 
Class 

William Lee Proctor 



This study involved the comparison of three moaes cf instruction. 
(1) lecture-discussion, (2) computer assisted* instruction— linearljL :• 

sequenced, and (3) computer assisted instruction— student sequenced. The 
sequence of the linear program is predetermined, *in that the student exerts 
no control over the sequence in vhich- the various concepts are presented. 
The student sequenced CAI program is designed to permit the student to 
select his own path through the material. In addition, this latter 
presentation will allow the student to recycle through th'e various 
subsections of the course. 

The subject matter taught during the experiment was selected from 

an education course pertaining to curriculum development in elementary ahd 

secondary schools. The CAI tiograms were taught via: the IBM 1500 system. 

* • ' . 

The investigator served as instructor for tlie lecture-dxscussion grou|>^ and 

all class sessions were tape-recorded. 

Variables investigated included learning acquisition, retention, 

instructional time, attitude, and essay test performance. Student response 

records also permit an analysis of the extent to \diich students in the 

self- selection of sequence treatment deviate from the 'linear format. 
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PrGXiinin 3 .iry snstlysis of xGsuXts indicstGS tlickti tiliG CAI**studGTit 



sequenced group performed best on both the post and retention tests. This 
group also xeflecbed the highest positive attitude to»ard the method of: 



instructiOT^'fe^lo^d A At this point, the statisticaL^ignificance of ;the 



results has not been ascertained. 



B 



An Investigation of the Application of Comput er Assisted Instruction 
and Information Retrieval Systems to Academ ic 
Advising in a Junior College 



t ' 



Harry V. Smith 



■ ^ wTf ^ • #’1^ 






This -is a study of. the application of Computer-Assisted Instruction 
merged vith a process pf information retrieval to the problem of academic 
advising of junior college students. The study will investigate four areas: 
a) information suitable for inclusi 9 n in an automated advisory program 
will be determined by interviewing both students and faculty advisors, 
(2).,a.progr^ for.the automation of certain aspects of the advising function 
will be designed and implemented, (3) the efficiency of the student-machine 
interaction will be studied, and (4) the adequacy of the automated system 

■ “ - .,»v ^ 

will be, evaluated. 

The evaluation will be on the basis of the following operationally 
defined criteria: ■ ■ ■ 

' jja .. .Jr* • < ' 

1 . Acceptability . r-The .students' attitude toward the automated 
process is not significantly less than their attitude toward 




the human advisor. 
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2. Worthwhileness . —The attitude of a faculty jury towatd the 
automated process is not significantly less than their attitude 
toward a human advisor. 

3. Systems adequacy .— The number of faculty jury requests for rele- 
vant information which cannot be incorporated in the systems 



program, will not be significant. 

4. Systems efficiency .— The number of trouble signals on' the part 

of the student and/or the number of sign-offs due to equipment 
* « ■ * “ « ‘ 
failure will not be significant. 

■' - ,’r. 

The primary aim of this study is to provide a miethod of information 
retrieval from a student record and course descrxption data base, and to 



merge it with a CAI progran which provides a logical dialogue, in order t6 

* N_ »■ 

automate and simulate certain aspects of the human advising fiinction, and 
to demonstrate that such a system is worth\^ile, as defined hei:4in, in an 
operational sense and acceptable, as defined herein, in a human factors ' ' 

sense. 



The developmental sample will consist of ten students and ten 
faculty advisors"irbm Tallhhassee 'Junior eollei^ (selected b^iihc chief 
counselor) and will be questioned in tape-recorded interviews to determine 



the information to be placed in the automated system.. After the interviews 
are completedo^tihe information on the tapes will be listed and tabulated. 

An attempt will' be. made to include in the automated system all. information 

which is requested by two or xiore students or advisors. 

A record which includes information such as students* 
grades for completed college courses will be stored on the systems disk and 
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will be the key for the logic flow and retrieval. Codes retrieved from 
the student record will be usad to branch the student to that portion of 

the CAI progrsn related to his academic goal. 

After the syste~c program is complete and operational, the author, 

a faculty advisor, and a student ^d.11 trace all branches of the program 

in order to debug it. 

• J M " 

A se‘>ected sample of 50 students will be chosen to participate 
in the study. All responses by the students during the dialogue will be 
written on disk and will be available for later analysis. A jury of six 
academic advisors will interact with the system as though they were students 
and, along with the student population, will complete an attitude question- 
naire designed to assess the adequacy and acceptability of the system. A 
detailed analysis will also be made of the nature and quality of the 
students' questions which are not satisfactorily answered by the automated 

system. 



C' The Interaction of Examiner Attitude with Praise and Blame. 

• ■ 

Ted M. Wilson 

" "praise and blame have' been found to be •stable and effective 
reinfofcers in the acquisition of motor skills (Kennedy & Willcutt, 1964). 
It^fihs ai^o been fdun<r thdt th\k-^-effect of these variables- may be greatly . 
altered by the examiner M ataraCtreristics such as race and attitude (Allen, 
et_al.', 1966; Kennedy & Vega; 1965; Kirschenbaum, 1962). 
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In the present study, an IBM 1500 Computer ^Assisted Instruction 
(CAI) system was employed to standardize the. administration, of praise and 
blame and to control the examiner's attitude as conveyed to the ^s. 

One hundred and twenty-eight university sophomore and junior 
psychology students were subdivided into. 16 groups of 8 £s each. Two 
sets of 16 oddity discrimination problems were utilized to assess the 

g • » » * 

effectiveness of four levels of verbal incentive (praise, bl^e, neutral 
statement, nothing), and four CAI examiner attitudes (positive, negative, 
neutral « nothing)... All combinations of incentive and attitude were employed. 
An attitude statement randomly accompanied 8 of the initial 16 problems. A 
verbal incentive statement was. then displayed to the student. This was 
followed by. the representation of the 16 oddity problems. Response latencies, 
errors, and ..posttest attitudes toward CAI functioned as dependent variables. 

Although the data analysis for this study is incomplete at the 
present time, it is hypothesized that positive examiner attitudes will 
enhance the effect of praise, and that negative attitudes will retard them. 
The converse is predicted for the influence of blame. 




IV, individual A^. GROUP, DIFFERENCES IN LEARl^ING UNDER TWO 
-DIFFERENT* WODES OF COMPUTER- ASSISTED INSTRUCTION 



Aaron Bauldree 

■ .1 .V 

The purposes of this -.study, .^er a . (a>^ determine if two CAI macro- , 
adaptive instructional strategies, .would, produce differential mean achievement 
and (b) to investigate possible lef^wer-variable by instructional strategy . 
interactiopis which might:;. occur for .twp different macro-adaptiva approaches.^ 
The first; strategy consisted primarily of a series of multiple chQice test 
questions related to the class text. The second macro- adaptive approach 
consisted of .a series of lecture-like presentations. 



Fifteen* Guilford-tyjpe ability factors thought to be relevant to success 
in either:iOne;or.both of the instructional strategics, were. rank ordered by 
twenty-six pre-trained judges according to their. pleyance. for predicting 
success in each of the instructional strategies,. . Seyen factors were rated 
in .the top f iye categories for one or both instructional strategies . . A 
battery of eight tests measuring the seyen f aqtprs, was administered to ninety- 
five Florida State University students who were enrolled in courses in which 
the instructional materials utilized in the investigation could be made a 

regular part of the instructional process. 

Students were randomly assigned to treatment groups. Each student was 
given the equivalent of three class periods of instruction; that is, each 
student was given the regular textbook reading assignment and then, rather 
than reporting to the classroom for the customary lecture, he reported to the 
CAI Center for individual presentation of the learning materials. After the 
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three period^ of instruction by CAI, all students returned to the classroom for 
a posttest designed to measure acheivement in the CAI sessions, 

TABLE I 

‘ THE 8 FACTOR ABILITY TSS't AND THEIR 
- ■ . . . RESPECTIVE CODES AND FACTORS 



tests 

• • » • m , ^ 

1. Sentence Order (o3b) 

2. Memory for Word Classes (o4A) 

3. Addition Test (N^l) 

4. Division; Test (iJ-2) 

5. WideKRange Voc 2 ^^.ulary Test (V-3) 

6. Ob:ject^Number T^st. /M^.-Z) 

7. Necessary Arithmetic Operations (R-4) 

8. Hidden Patterns Test (cf-2) 



Factors 

Convergent Production-Semantic 
Systems (NMS) 

Memory-S 3 nnbolic-Classes (MSC) 
Memory-Symbolic- Implications (MSI) 
Memory- S 3 rmbolic- Implications (MSI) 

I _ 

Cognition- Semantic-Units (CMU) 

.Memory-Symbolic-Relations (MSR) 

Cognition-Semantic-Systems (CMS) 

Convergent Production-Figural- 
Trans format ions (NFT) 



Mean differences for the two macro-adaptive instructional-strategy groups 
were determined for (1) each of the eight ability factor tests, (2) achievement, 
at each of three different stages of learning, (3) time spent reading textbook 
assignments and (4) time spent on CAI. No significant group differences were., 
obtained for any of these variables. Thus, one could conclude that different 
Instructional strategies lead to the same level of group performance on criterion 
measures. The situation with respect to individual performance and the ap.titude- 
treatment interaction '"i‘s more difficult to interpret. 
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Differences between individual' simple xegression coefficients, for eabh 
instructional sf'.-tegy grbup relative co each ability factt'r were computed, the 
results show that six of the seven ability factors used in the study produced 
significant interactions at some point during the learning period. The six 
factors which produced interactions aire Cognitidn-Semantic-Uhits (CMU), Cognition- 
Semantic-Systems' (CMS) T^feniory-Symbolic-Relations (MSR) , Hemory-S 3 ^oi-ic-Implications 
(MSI), Convergent Production-Semantic Systems (I^S), and Convergent Production- 
Figural-Trans format ions (NET). However, factor CMU was the only factor that 
produced consistent results throughout the entire experiment. The failure of 
the other factors to produce consistent results might be attributed to a change 
in the ability' factor patted that could be used to predict success at different 
stages of learning as reported by Fleishman (1957). If this is the case, guide- 
lines for adapting instruction to individual differences can be established only 
by studies that extend oiret the entire period of instruction for whicn achievement 
is to be predicted. . 

Correlations between amount of time spend reading textbook_ assignments and 
achievement were negative and significant for Group I. For Group II, the 
correlations tended to be negative but were not significant. 

Correlations between amount of time spenS on CAI for Group I were close to 
zero, not significant and tended to be negative. For Group II, the correlations 
were negative and significant. 

Thus, it appears that low achieving students in Group I spent more time 
with textbooks than did high achieving students. In Group II, the high achieving 
stsidents spent less time on CAI than did the low achieving students. Both of 
these conclusions are due in part to the actual instructional strategies involved* 
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In sumaary, the folloving findings resulted froQ the experiment: 

1. The mean differences for achievement of the two groups were not 
significantly different. 



2. Six of the seven factofs produced significant interations at some 
point during the learning period. 

3. The only factor which produced consistent results 

entire learning period was the Cognition-Semantic Units (CMU) factor. 

4. Correlations between time spent reading textbook assi^nts ^ 
achievement and time spent on CAI and achievement indicated that the 
diagnostic practice instructional strategy stimulated more 

reading aiDong low achieving students than did the programme ec u 

Strategy. 
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--cumcimim evaluation via cai for the 

ICTERM2DIATE SCIENCE ' 

- CUmCULIJM STULY^ ‘ 

- I- . . Walter Dick 

... : . 

-i Florida University 

ISCS, the Intermediate Sci-ence Curriculum Study, is a project 
devoted to the development of a self-paced, process-oriented set of 
science curriculum materials for use in seventh, eighth, and ninth 
grade classrooms. The project is directed by Dr. Ernest Burkman, and 
is staffed primarily by faculty members from the Department of Science 
Education at Florida State University. This resident staff is joined 
annually by a summer writing team of subject matter specialists. The 
immediate goals of the project are to develop a rationale and a set 
of specific objectives for science instruction, to write the instructional 
materials, and to study the impact of these materials on a national 
sanple of junior high school students. An additional goal of the project 
team is to use these instructional materials in long-term research 
studies which will be conducted in classroom settings. 

During the summer of 1966, the materials for the seventh grade 
were written and, subsequently, evaluated during the following winter. 
During this past suinner, the seventh grade materials were revised, and 
the initial set of eighth grade materials was developed. Next simmer, 

^Paper presented at the American Educational Research Association 
meeting, Chicago, Illinois, February, 1968. 
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materials will be prepared for the ninth grade. Two n^hods have been, 
and are being, employed to evaluate these materials:, large-scale 
field testing and computer-assisted instruction. Approximately 5,000 
students and 50 teachers used the materials in various schools aiound 
the country last year, vhile over 11,000 students ^d 100 teachers are 
using them this year. The CAI evaluation has been conducted with 16 
students during both years. i : : 

What is the rationale for the use of CAI as a curriculum evaluation 
tool? '“Perhaps the outcomes of both CAI and large-scale field tost 
evaluations can oest be described in terms of two, somewhat overlapping 
circles. - Both types of evaluation provide general measures of the 
effectiveness of instruction and an indication of the more noticeable 
rough' spots in the materials. Field studies provide, in addition, 
information on the logistics of implementing new materials and 
management problems encountered by -a large number of teachers. In 
addition, they provide demonstration centers in \diich other teachers 
and administrators may view the materials in use in the .classroom. 

The unique contribution which CAI is now making tO; curricnlum evaluation 
is that, for the first time, fine-gijained data can be obtained during 
the formative development of a curriculum; data which can be utilized 
to improve the instructional materials. It provides information on 
the learning difficulties of the individual student as he is sequenced 
through the learning materials. This data can also be sorted in such 
h way that group performance on particular concepts can be investigated. 
Therefore, because of the ISCS interest in the evaluation of their 
instructional materials via CAI, a joint project was established with 
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the CAT Center. We, in the Center, have three major interests in 
this, project \^ich may best be stated in the form of questions: 

Q) What are the procedural requirements for successfully 

implementing an entire year*s curriculum in a given 

. . - “ " . * *- * • 

area? 

•J' - : . . . . . • 

(2) Will a generalized data management systm provide data 

\diich can be effectively used by curriculum writers? 

• « - « - «. - •- - 

(3) What is the actua.1 cost of this type of evaluation? 

The remainder of this paper will be addressed to these three questions. 

In order to obtain the evaluation data, the seventh grade text, 
\^ich is being used in the classroom, has been prepared by the ISCS 
staff for CAI presentation follc^ng essentially a programmed 
instr^tion, linear format. The course also includes numerous 
excursions to which students are branched for both remedial and 
enrichment instruction. Every effort has been made to simulate 
OAI learning experiences similar to those which other student.* are 
experiencing in the classroom, and therefore feedback is provided to 
the CAI student only when similar feedback would have been provided 
if he were studying the regular instructional materials. 

The students who are serving as subjects in this project have been 
carefully selected from the seventh grade of the Florida State 
University School in which the ISCS materials are concurrently in 
j 2 se. These students represent, in terms of a measure such as I.Q., 

- ■ • - . : • V ■ 

-a range of abilities which is likely to be found in most seventh grade 
science classrooms. The I.Q. range of this year's students is 85 to 

I 

I 

o 
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Rather than receiving the ISCi! materials in their regular 
classroom, these selected students come each momiti^ to the CAI 
Center and progress at their own rate through the learning materials. 
Since laboratory experiments form an integral part of^ the XSCS 
approach, the students are directed, by their prograsi, to go to the 
laboratory area within the CAI Center, to carry out these activities. 
When they return to their terminals, they are asked questions about 
the experiment, and are required to submit the data which they have 
obtained . 

There is a graduate instructor with the students at all times 
^en they are in the Center. It is his function, as he simulates the 
role of the classroom teacher, to encourage the students to arrive 
at their own solutions to the inevitable questions which arise. Only 
as a last resort does he lend direct assistance.' Hie ijociimentation of 
student problems by the instructor and how these problems are solved 

" t 

are a valu^le supplement to the response data. The proctor comments 
are now entered immediately and diiectly into the computer systan and 

are available for retrieval later. 

- ■ - • • 

Although the major purpose of this CAI activity is to gather data 
for the revision of the instructional materials, it is obvious that 
some comparative analysis of the performance of CAI versus ISCS class- 
room students was required. Several times during the past academic 
year and, continuing this year, similar tests ha\e been given to all 
the ISCS students in the University School and in the CAI Center. 
There h^ been no significant difference in the performance of the 

I i.*L. . , 

two groups. In addition, none of the students have requested to be 
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tranQ^fejrred from the CAI group back to their classroom. In fact, 
following approjoimately 120 hours,.of .^ps true t ion last year, the students 
. were still quite enthusiastic _^out their experiences . Their responses 
to a questionnaire. further reflect. this very ppsitive attitude^ 

!• . We: at the-CAI :Ccnter, therefore, feel that the answer to our first 

iquestion.AS,"Yes, we have evolved an understanding of the procedural 
* -requiremqpts- for implementing an alrea4y. existing set of materials on 



-vour -PAl isystem.'* . 

Our second question dealt with the feasibility and acceptability of 
. a generalizedj4ata. management system. One of the major projects over the 
= past^.twp. years in our Center bps .been the development of a series of 
prograns which.-yrould form a .system for reducing and analyzing the 
. thousands of r student re.sponses \diich are recorded on magnetic tape. 

It is one thing to develop programs which patisfy the needs one or 
. :two in-houae:cProj.ects, apd .. quite another to develop a systOT ^ich will 
: -prod^e satisf^tory data, formats for all users. 

.: ^...;.v:The basic record \jhich is .written each, time the student responds 
to a question on our 1500 system.includef, among other thxngs, student 
. and jcojiyse: identification, . date, progr^er-genepted idptifiers of 
:..:.:the, qpeptiop which.- is.. asked and fhe. response ^diich.is pde by the student, 
,::.responSrft-latency,-,and.the nume.rical values of all the counprs and 
.i:-awitchfts>r Vety, briefly, our d^ta managemep programming system first 
reduces and compacts these recordings. It. then sorts out the responses 
. by each. student within each coiurse. . This is done on a daUy bpis. 

Periodically, these, sorted, tapes are merged into a student history fxle. 
i This .file, therefore, .contaps -the respons.es p all students to all 




3Y 



items to that date. At/.thia point, the file can either be printed, 
which would indicate..the path of each student through the material, 

or it can be sorted further by item. Our print-out of this item sort is 

, . 

in the form of a summary t^le which includes^. the number and percent of 
'the students x^o match each response option for. .each item. It also 
Indicates the median latency- fop each group of responses and the 
standard deviation. This is the type of information xdiich was provided 
to the ISCS staff. ..An additional program produced a matrix of student- 
by-question responses xghich indicated the general pattera of answers. 
Student -generated, schematic diagrams and data tables, as well as 
proctor comments, were also provided. All of this information was 
integrated by the ISCS staff and, subsequently, provided to the 
summer writing teapcf. .. ;r. - 

This information x^as used to a certain extent by the summer 
x^iting team, but..their general comments indicated that there was too 
much information, that it should be more highly summarized. They 
further indicated the desire to be. able to easily locate and cross- 
reference data on concepts that appear in a number of locations in 
the materials. 

Oh the basis of this ^perience,. we went b^k to the draxd.ng 
board' this past fall. . The system xdiich we. had. developed was necessary, 
but not sufficient for the specific needs of ISCS. Txjro major changes 
have been made to date. The first has been to add a select-and-delete 

X. 

capability to our data manugement system for clustering item information, 
^hia Can best .-be understood in terms of the .item or questxon identifi** 
cation procedures. The programmer can label these items in any way he 
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sees fi'tj iTlthin certain restrictions,. The ISCS staff has coded all of 
their items. with respect to their content; for example, work, speed, light 
ehetgy,: etc., and with respect to the process involved in the item; for 
example, operations i^ith decimal .numijers, hypothesis formation, making 
observations, etc. With these l^eT^ identifiers, it is now possible «o use the 
Select function in order to extract the summarized item, data covering 
■^ "sp^ific topics or processes ^di^h occur fn a variety of locations in the 
• instructional -materials. Using this same technique, it is also possible 
to select out those items which had median response latencies of greater 
than soTO'-nomber of seconds or those items which, were correctly answered 
by less than some given percentage i. 

The second major change has been the decision that the data reduction 
3nd'^ summarization ■ beyond- that already described, \7ill be carried out by 
the resident staff of the ISGS project and not the members of the summer 
writing team. On the. basis of this. and other.p.p.ojects, it seems unlikely 
■th'dt' sub ject-matter:. specialists will ever be willing^ or have the in- 
clination', to sc an- vast amounts of data and. then, rewrite their materials 

on the basis of this information. Therefore, the ISCS staff will now 

1 1. r* 

be responsible for the task of converting the selected it.em and student 
' "' data into sound recofimendations for material revisipn. If the need arises, 
the -data- Will be. available to back up such recommendations but it will 
not Simply' sit unused on the writer’s desk. 

■ Our- answer to the second question ^diich related to the viability of 
*k gehef^ized data management system must be one. of restrained optimism. 

The development of.. such a syst.em is obviously an evolutionary process 
which should become more and more refined with each succeeding interaction 
* between the data and user. 

I 

I 

o 
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Our final question relates, to the costs of this type of CAI 
evaluation. Everyone is conscious of the current, rather I'.ight fiscal 

- IT 

*• m * ' 

situation. No one is more aware of this condition than the users* of 

S ^ 

stand-alone CAI systems'.' Ne have been required, by our University, 
to establish the rates for the use of pji.r CAI equipment based upon 

* * t 

“ ^ - r * “ “• « 

the same type of procedures which ar^ recommended by the Federal 
Government for usage by conventional, batch-processing computer 

• “ - " * ■ T ^ 

- • — .i . , - ■ 

centers. 

In order to generate the costs for implementing the ISCS project 
materials, we have not Simply, multiplied our systems rental cost by 

the percentage of usage time chargc^.fe to ISCS, but have included 

“ ~ " - 

such items as unit record equipment rental charges., laboratory re- 
modeling costs for accoramodatipg our new equipment, maintenance and 

- j ' • • ■ • 

janitor’"! services, certain portions of faculty and staff salaries, 
general office expenses, and computer supplies. 

The total costs for the ISCS project, therefore, reflect all of 
these items as they enter into our charges for both the central 
processing unit and terminal usage. In addition, the costs of 
the three graduate assistants, a coder \dio enters the materials into 
the system, and one-fifth of a faculty member* s time have also been 
added . 

The sum of these charges for the current nine-month phase of the 
project, using the 1500 system, is $34,865. It is of interest to 
note that this figure of approximately $35,000 represents a cost of 
about $12 per instructional hour. However, only $2.13 of that $12 
goes for student terminal time usage. The remainder covers the cost of 
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material preparation, proctoring, and data reduction.’ These figures 

should, in no way, be construed as an argument for the economic 

feasibility of placing CAI terminals in all ohr schools. The field- 

is a long way from the $.40 per classroom hour that we purportedly now 

spend on our children. The costs are interided to show that CAI does 

provide a rather economical and feasible approach to the gathering of 



fine-grain evaluation data. The term "economical" is used to refer to 
the fact that only a small percentage of the funds usually required 
for a large field study could be diverted in order to provide a 
CAI-based evaluation. One can also expect that, after two or three 
cjreles through this process, there will exist not only a much evaluated 
and revised set of dassroom instructional materials, but also, as a 

by.prpduct, a quite reasonable CAI instructional program. 

» ■ ■ 

i: ' • . . 
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COMPUTER ASSISTED INSTRUCTION 



IN SOCIAL WORK* 

. t 




Walter H. Ehlers, D.S.W. 

Professor of Social Work - , 

. Florida State University 

COMPUTER ASSISTED INSTRUCTION, TECHNOLOGY AND VALUE ’ .. .. 

The old cliche - "there is nothing new under the sun" - needs, revision^. 
There is something' new called computer assisted instruction and educators, 
ready or not, will have to learn this new technology and its terminology. 
Computer assisted instruction is also known as computer based . ins tru.qtipp. 
and computer augmented learning. Whether you use the letters CAI, CBI or 
CAL you are talking about' essentially the same thing, the use of * a .computer 
to handle some of the elements 'of teaching. The word some is used.,because 
most specialists in this area do not see the computer as ever, replacing the 
teacher either at the grade school, undergraduate or graduate .leyels of 
education. Instead, it seems to be generally agreed that certain .functions 
may be safely assigned to a cbmputer, e.g. it may serve as an e^fcellent way 
to instruct in terminology, to give extensive practice in basic skills.,, .and 
to teach problem solving. 



Duncan N. Hansen. 
UniVersityj Tallahassee, 



Computer-Assisted Instruction at Florida State 
1967, p. 5. (Mimeographed) 



*Prepared for presentation rt the Annual Program Meeting, Council 
on Social Work Education, Minneapolis j, January 24, 1968 
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Technically what we have in the ccmouter is a mechanized tutor 
capable of con.;ronting the student on a one to one basis, a goal that may 
have been achieved in the case of Mark Hopkins on one end of a log and 
the student on the other- The tutorial .relationship is rather rare in 
modem education; instead we find ourselves teaching larger and larger 
classes. In some of our' universities dadses of 1,000 students meeting 
in a large auditorium for a lecture in elementary psychology or sociology 



are not unknown. In graduate education we have the expectation, if not 
always the reality, that we will have some small seminars tut a goal of 
tutoring students is outside our reacti. The computer changes all that 



and offers the possibility of endless arid tireless tutoring* Using CAI, a. 
question may be asked of the student, and a variety of possible responses 
may be programed, "fe instructor allows for the 'cdrrecf response or 
the sought after response,’ but allows for interaction also 'with the student 

giving partial or incorrect responses, ' . 

The infinite patience of the machine then may be utilized by- the: ■ :l 
instructor to do the chore work of endless repetition and discussion. ■- 



until the desired learning has taken place. Instruction in terminology is,, 
as previously stated, an excellent use of the computer, but what about. . 
practice in basic skills and problem-solving? 

bilities of using the computer to simulate siti*ations needs description. 

■ ' ' * • - . * .1 . „ . i «. 

The capacity of the computer to store in its magnetic memory millions ’ 
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of characters, sroups of which may be used to form programs, permits 
us to devise simulations of actual situations 'irtiile allowing for a wide 

variety of exceptions to be posed for the student. These exceptions 
can be incorporated into .a problem-solving and decision-makmg matrix. 

As in the games strategy and gaming techniques the student is confronted 
with facts and with choices. His selection of the right or wrong answer 
is imoediately confirmed making the learning process a dynamic current 
experience. Although much more could be said relative to CAI capabilities 
from the technical standpoint, two more advantages ought to be mentioned. 
One, that CAI is basically programed instruction with the added capabilities 
of a CAI system to imaediately evaluate student responses automatically, 
something not possible with a textbook or teaching machine. Two. the 
student may be branched into learning programs instantaneously by'a . 
computer decision based on the cumulative records being kept on the 
student’s progress. In the latter case it is possible for the instructor 
or author of a program to set up a series of counters which check 
student's failure rate, and at some previously decided critical point 
branch the student to additional learning material or recommend that 
the student review certain readings or even come back at a later time 

and take a prescribed learning program* 

From the instructor’c point of view then the computer may be seen 

as a tool, just as a book is a tool, which he may use as a willing slave. 
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The instfuctor may then be available to students for the intellectual 
jousting which is so much more important but often is neglected because 
time is simply not available. He may also pose the question of whether 
the educator using the 'computer instructioii capabilities might then be 
released to do the thinlcing and writing si> ' important to the academic 

coomtmity. 

COMPUTER ASSISTED. INSTRUCTION AT FLORIDA STATE UHIVERSm 

Research and development of computer based instruction was begun 
at Florida State University in 1964 by the Institute of Human Learning. The 
research led to the.e.stablishing of a computet assisted instruction center 
on the FSU campus, which went into operation April of 1965 doing rese r 
on the prospqcts of CAI and developing programed materials for instruc- 
tional purposes.^ An introductory., c.^rse in collegiate physics, appUed 
statistics, computer languages, chemistry, basic reading and mathematics 
for adults are examples of some of .the CAI courses offered in addition to 
the social work basic concepts course to be described in detaU later.. 

In opsaratiqn at the center are two computer systems, the IBM 1440 
and the IBM.ISOQ. The, IBM 1440 system components are, a processing 
control unit, three disk memory units, a magnetic tape unit, a console. 



Dtinrcn N. Hance*'j P* 




a card read/pcach device, a printer and a data connunications system. 

The last item provides input and output by means of the 1050 \diich resembles 
and works like a typewriter. This system also allows for random access slide 
projection and random access tape recording equipment. Students and authors 
enter and receive information within the Central Processing Unit by 



using the 1050 terminal. 

In September 1967, the CAI Center added a new computer, the IBM 1500 
which essentially is the same as the IBM 1440 but with a vidw screen 
similar to a television screen and with a light pencil attached. Students 
see an entire question and the choices of possible answers. Using the light 
pencil attached to the terminal, the student marks his answer and Immediately 
is advised as to the correctness of his choice. The IBM 1500 therefore is 
a much faster machine than the IBM 1440 which types out all instructions 

f • ^ 

and responses. Both machines, however, have their advantages and dis- 
advantages and should be used selectively according to the need. 



RATIONALE FOR CAI PROGRAMS IN SOCIAL WORK 
In the srring of 1966 preliminary xsork was begun with fifteen 
graduate social work students at FSU to determine the feasibility of 
computer presentations of behavioral science, knowledge considered pre- 
requisite to advanced social wrk education. For the feasibxlity study, 
programs were written iiv -.sociology, psychology, child development. 



46 



and Freudian analytic concei^ts. Faculty members of the University, 
(Psychology, Sociology, and Hoiae and Family Life) were asked to 
review the programs, to suggest changes or corrections, and to 
recoDBiend new directions if necessary. The resulting programs were 

the ones used in the feasibility study. 

The interest and enthusiasm of the first fifteen students for the 

CAI programs was enough to suggest further exploration. The. fact 
that the students were volunteers in the pilot program and therefore 
were singled out for special attention was considered to have had a 
possible "Hawthorne Effect" in respect. :to the expressed interest and 
enthusiasm, Hon^ever evidence of learning was strong enough to 
warrant a larger evaluative effort which was .undertaken in September 

1967 with 113 members of the first year class. 

Feasibility having been tentatively established, the second 
evaluative effort had as its objective the use of the four programs as 
screening or diagnostic devices for the development of student profiles. 

A second part of the objective was to lead students weak in any of the 
four programs through a learning program or learning loop especially 
designed to strengthen their basic knowledge. Material for the four 
programs in psychology, sociology, child development and Freudian 
concepts was selected on the basis that (1) social work requires a basic 
in the behavioral sciences, and (2) that although no consensus 
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exists regarding specific content prerequisite to professional training 
yet there is enough agreement in what we assume students should bring 
with them in the way of academic background so that we can at least 
select the areas of knowledge to be tested.. At this point in the research 
the test questions have not been validated with control groups since this 
is projected for the next step, but they do seem to offer a reasonable 
approach to the question: What do students need to know upon entrance 

into a graduate school of social work? 

The fact that students come from a wide variety of acadeniic 
backgrounds makes it even more imperative that we do not assume that 
students have had adequate preparation enabling them to understand 
basic concepts and terminology of’ our first year graduate social work 
courses. This fact presents us with a challenge which might be met 
with pencil and paper tests. However such tests lack the kind of 
immediate feedback for student and teacher possible in computer 
assisted instruction- But, even more important than testing the 
student is the necessity to institute immediate remedial work in his 
area of weakness. This of course can be done instantaneously in CAI 
and offered the final ccxapelling reason for experimenting with CAI 
programing! Additionally, this reason satisfied the question of, 

'Why one more testing program for students already accepted into 
graduate social work?" with the answer that students using CAI can be 
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assisted in a ««aningful way to an understanding, of curriculum content . 
necessary fir rapid advancement during the first eight weeks of Braduate 

school. . . ■ . • ■: • > -ji 

RESEARCH DESIGN AND DISCUSSION Oe FINDINGS 

This second evaluative effort (labeled Phase II) involved 113 first 
year graduate studdnts entering social work in September 1967. Taking _ 
the CAI program was made mandatory for every member and only 
blind student with- a ninety per cent visual handicap was excused. The 
rationale- for this action was to determine- total student reaction to 
computer assisted' instruction. -Phase III .^ich, will begin September of 

1968 wilt -involve a fifty-fifty split of that -fir^t ye.ar class into 
expeciaantal-^d coritfol groups.- Assignments to either group will be ^ 
made using a table of random numbers. A post test of the 

will then be conducted to -ascertain possible differences bet^en students 
taking or not taking computer assisted xnstructxon. 

The'priK:edures 'for the September 1967 testing were rather simple. 

A list of all f-ifst year students with their .student numbers was given to 
the CAI center. - Students phoned the center apd, t^de tteir own appoin ^ 
ments based on their.- available time and , also ^availability of terminals to 

the computet. ' ' ’ 

• Each' of the- programs in the.-four areas ;Of .knowledge mentioned was 
designed to be between one to one-and-a half hours long. The intent here 
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was not to tire the student which eight be reflected then in poor responses. 

If time. was available and the student worked fast it was thought that some 
students might go on to a second program.' This in' fact was exactly what 

did happen. , 

Each of the four programs was designed with a test section which 
was given first. If the student did poorly in any area he was exacted to 
take the learning prograa, which in essence went back over the saae 
material but this tiro gave the student information which made it possible 
for him to learn the necessary facts about the subject, and to answer the 
questions correctly. The intent here was to design tests i*ich would 
reveal student weaknesses in the four selected areas, and then allow for ' 
remedial wppk using CAI in order to bring the student up to a level pre- 
sumed necessary, to understU beginning social=w<;rk terminology aid ' " = 

concepts. _ 

The programs, .although designed to be given to all students, allow fof 
both priv^y and ind^idual differences. 'Each student' «rics at a typewritet- 
like terminal which ip appearance and oi^rationii similar to an electric 
typewriter, and in a separate cubicle the tenninal. a desk and a' 

chair. Each room is also equipped with a buzzer system to call the prdctdt 
who is. always present to offer assistance in the operation of the terminal 
to handle technical problems. In order to awid a completely impersbhal’^ 
atmosphere the prograns were written with a touch of humor and at 
strategic places the computer asks the student ti., "Think of me as Ce&ge. 






The first time this remark shows up on the printed page students arc a 
little surprised but they do like the idea of the human touch and often, xn 
later class discussions will preface a statement with "George said..." 

This is usually good for a laugh from the test of the class. In fact, since 
the entire CAI program in social work is called CONSOWEL (short for 
Concepts in Social Welfare) the name George Consowel has been suggested 
and is under serious consideration for Mr. George Computer! Students 
can also rid themselves of some hostility at times when they can persona- 

lize the computer. 

The feeling of individual instruction is made possible by the nature 
of the typewriter terminal and the computerized programing. The student 
has the statements and questions printed automatically by the typewriter. 
Likewise, his own responses as they are typed become a part of the record. 

A question answered incorrectly or with an unfamxlxar response can later be 
examined by the instructor, who, on the basis of a review of the question 
may want to add the students answer as an acceptable one, or it may even 

cause the instructor to revise the question. The student’s copy of the 

material, called a "Print Out" is his to take with him and review at some 
later date. 

In order to further individualize the instruction the research design 
called for "counters" to be inserted after each question so that students 
would know at the end of each program how well they had done. For those 



receiving high scores the statement on the printout said, "Great, your 
responses indicate a good background in the «ea of psychology (or 
whatever subject) and you should not have any difficulty with this area of 
your social 'wbat studies." Other students whose count of incorrect answers 
was high, received .a statement directing them to take a learning pro- 
gram in psychology (or whatever) and to make an appointment at the front 
desk for this additional CAI material. It should be noted here that in 
using computer assisted instruction strictly In the instructional mode (not 
testing) it is possible to branch a student to a review of material or 
further explanation of concepts immediately, without needing to make 
appointments to take further instruction. However, .since Phase II ya® 
designed as diagnostic, a different approach had to be used. , ... 

The final component in this design was to request from the, computer 
a response report on each student so that a detailed examinatjion co . 

be made bf every response of the 113 students. From this, set of 
sheets, one could theoretically profile,studentd responses, individually ^ 
collectively against any known mean. der4.vcd from the questions in 
programs'. 



FINDINGS 

The results of the second evaluative study have produced a number 
of observations Uich are lifted below. The nature of the evaluation to - 
date has been observationai'"iuid although no numerical count was attempted,- 
in general, students reported that they "enjoyed," 'Vere interested," or - 
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"intrigued" by ^h'ci tAI p^ Most craimed that they "learned a lot," 

or that "the review v^ry helpful." Many claimed that the review 
factor alone’ was wofth tlie' time it took to -complete the- programs . Qther 
faculty members reported that students would quote from the- CONSp^lp^ 
programs citing" CONSOWEL as their refer«fics on a particular ppiut... Stu-. 

dents reported that they seemed to remember f acts.'lihat. were reip^PTPcd 

- i r ‘ * - 

in the programs. 

Perhaps predictably, the better studetfts seemed more enthusiastic, 
completed the' progrins eitly ind were anxious to discuss various qugs- 
tions which' had' occurred to 'them. ■ -The weaker students were -more fear-, 
ful, resisted making appointmdhts" and' found excuses for delaying completion, 
One surprise Was to find -that- wdmeh students' Were much slower in 
gettinV staried'on the programs claimed they were "busy" or .>t used..#o 
machine^" or whatever.’ At any rate the women students respon^d, slowljr . 
out of SOTii fear of failure or discomfort. After completion J^ver most 
of the 'TOmen'idiitted 'that their fears had been groundless and that .in ^qct 
they enjoyed tik'ing tSie programs. Only one woman .student was willing to 
admit that she "hated machinery," "didn't know how to type" and felt. ,, . 

uncomfortable throughout . 

In general, the findings showed that (1) the programs were well 
received, .(2) the students were interested enough to raise questions, (3) 
they remembered items in the programs well enough to cite them in class, 
and- (4) the students reported having learned. 




DISCUSSION 



The first part of t4«. objective for Phase-II^-to use the CAf. .programs 
for screening or diagnosiS-^7as successful. Students. were enabled to see . 
their Strengths or weakniesses as-a result of taking the programs, The 
student' profile component of the objective, however, was not included this 
time dut to programing problems and the fact that coiHiters were not . 
inserted sO that'k quick profile could be established immediately upon 
program completion; 'However . this- lack did not invalidate the-profiie 
concept.-'-fesponse print-outs on each student were made, but these are not 
iaoediately available either to the student or the instructor. The.se print- 
outs ire cumbersome and analysis is slow because of the sheer amount .of data 
Print-outs. -for example, for the 113 students resulted in a pile of paper . 
three feet high. The answer to this problem is to insert counters and 
ask only for the number ri^t or wrong and the percentages in each a'rea. 

The response print-out does serve an important function in another 
area; that is. the testing of the curriculum content. Statements .misunder- 
stood, or questions answered incorrectly two and three times by a suffi- 
ciently large number of students, indicates ambiguity, poor wording or 
other problems needing instructor (or another) solution. Fortunatexy. 
such changes or corrections may be made quickly and easily by going to the 
typewriter terminal and typing the revisions. The fact that such changes 
can be made relatively easily, means that programs can be kept up-dated 
and dynamic. Materials found to be inadequate or in need of further 
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development need not be kept in while waiting for the next edition, one 

of the mein problems wrth progrsmed texts. 

The second part of the objective for this evaluative study wai> to 
lead students weak in any specific area through a specially designed 
learning program. Here again, the original design called for stepping 
after the test" program and telling the student whether or not he was to 
go on to a learning program. Mechanical problems ajid lack of time to make 
-corrections for this group of students led t.^ a. decision to have everyone 
take both the test and the learning loop. Analysis of the student print- 
outs and discussions with a sample of forty students was selected as an 
alternative method of assessing- student reactions to the testing and 
learning locp combination. Subsequent analysis of the print-outs showed 
that weak students did in fact ir^prove their score.s in inpi>t case^ by as 
much as 100 per cent. This was not unexpected since the learning programs 
were written so that students should have been able to answer the questions 
if they had properly understood the preceding material. We may conclude, 
therefore, that this objective was met; that students* weaknesses in certain 
knowledge areas' could be profiled using CAI and that the learning programs ^ 
did assist them in being able to respond with the correct answers. 

CONSIDERATIONS FOR THE DEVELOPbENT OF 

SOCIAL WORK CURRICULA USING CAI . _ 

Recent advances in the field of computer and media technology have 
opened opportunities for the social .work educator never before available. 
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This new technical knowledgj comes at a propitious time, siace it coincxdes 
with significant developments in the behavioral sciences. Research in this 
area, much of it government funded, is resulting in quantities of data and 
resultant theoretic conceptions rich in implications for social work cur- 
ricula. Computer based instruction and computer simulations in decision 
making provide ideal pedagogical vehicles for implementing new dynamic 
curricula. The flexibility of CAT is such that other media (filids, audio 
tapes, slides) may be integrated into the programs resulting in a highly 



effective approach to the student. Results of a program in Physxcs, e.g., 
being conducted at Florida State University using this type of xntegrated 
media is producing some significant data. In tests given to experimental 
and control groups, the CAI-participating students performed approximately 
ten per cent better on all four, one-hour exams. ^ Hew well students perform 
after CAI is still under investigation. The CAI Center presently has eight 
curriculum projects in various stages of development; analysis and evaluatxon 
of these projects is designed into the .research. At FSU the question 
whether social work curricula may feasibly be designed for CAI xs stxll 
being tested and will await more sophisticated research being contemplated^. 

in the months ahead. "• 



^Duncan N. Hansen and Walter Dick, S emiannual Progress R^ jg r t 
January 1, 1967 through June 30, 1967, Tallaliassee, Florida Computer- 
Assisted Instruction Center, Florida State University, p.41. 
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In order tc properly design the programs or software that constitute 
the teaching materials of CAI, social work educators may begin now to thxnk 
of the following: (1) What subject matter is basic to the field; (2) What 

skills are basic to problem-solving., diagnosis, interviewing, therapy; and 
what -attitudes, values, understandings, emotional sets are basic to the 
profession. After establishing these components, the next process would 

naturally involve testing for validity. 

The necessity to move forward tauidly in the development of CAI in 
social. work education is related to manpower needs, insufficient supply of 
teachers, length of tis^ needed in the M.S.W. progrJca. Educators are 

4 

saying that "programed learning is here to stay," and that educators 
would do well to accept that fact and write the kind of instructional 
materials required Social work erucators .may be presumed to be included 
in this Simmons to action. P. Kenneth .Komoski, president of the Center for 
Programed Instruction, New York City, reminds us that,". . .with programed 
instruction -we have a tool which makes possible a rigorous, objective 
analysis of what happens when-. we. instruct.", and better yet, it helps us 
identi^. the obsolete by showing uSj objective evidence of something better. 



\awrence M. Stolurow, "Lets be Informed on Programed Instruction," 
Phi Delta Kappan, March 1963, p. 257. 

\bid. 



S. Kenneth Komoski, "Programed. Instruction 
Phi Delta Kappan, March 1963, p. 292 :>r- 
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Komoski relates that when he first started in programed instruction 
anc teaching machines^ he queried ‘psychoW^^^^ 

the field; •'What are the tilings thai can Sd "cdhnot'b programed 'for ^a 
teaching machine?" "It 'wasn’t' long before I learned ::he stock feply of 
my psychologist friends: "This is a question that can only be answered 
experimentally; experimentation and time will tell what can and cannot 

be programed." 

But, there is no denying the £act that anything social vork educators 
do at this point will be very much in the nature of pioneering and they 
will frequently find themselves in dark forests in which dweji the 
ubiquitous, "I told you so'sl" 



problems 

What are the problems? A few of the major ones include: (1) Identi- 
fication of the significant concepts which must be taught in the theory 
and practice courses; (2) Submitting the concepts to the scrutiny of 
colleagues; (3) Writing curricula suitable for programed learning; (4) 
Adapting the curricula to computer based instruction and allowing for the 
use of additional media along with the computer; (5) Selecting learning 
situations which may be simulated and which can be taught via CAI and 
related media instruction; (6) Adequcte research funds-because the 
development of new curricula will take a lot of time and cost a great deal 
of money. Mot mentioned before is the need for access to a CAI system. 
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At the time of this writing, only eight universities have operating CAI 
programs, but this should not deter anyone from moving forward. Hardware 
is not the problem; but the programs themselves, the software, are. 

' \ 

. « ^ - 




AN II^''‘2CT!3ATICN CF CC£'IPUTZH-3AS:^D 
GOONC^ TllGTING 

Dr. Duncan N. Hansen 
Florida State Urdversity 

Introduction. The sequential model of testing as opposed to the 
conventional cumulative tes* model directs the examinee to subsequent items 
on the basis of his prior responses, in operational terms, as the examinee 
completes a test item, the outcome is immediately evaluated, and this infor- 
mation becomes part of the test history file upon which a decision rule is 
applied in order to determine the next test event. Obviously , no examinee 
attempts all the items in the sequential test, but rather responds to any of a 
variety of combinations of .cems. This paper will concern itseK with the 
empirical findings of sequential testing as it affects test reliability , validity , 
scoring procedures, and the pragmatics of test administration. 

The availability of an interactive computer system and its capability 
for response analysis, record-keeping, and contingent branching is an integral 
component within our approach to sequential testing. While interactive con^wter 
svstems resolve the logistic problems of administration, they raise new consid- 
erations relating to examinee’s adaptation and attitude towards the electronic 
equipment, the type of testing, etc. Moreover, new test construction techniques 
and preparation factors like computer coding, operational costs, etc. , have to 
be considered if a reasonable documentation of the feasibility and cost effective- 
ness of con^)uter-based testing is to occur. 
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We turn now to a more specific background review of the psychometric 
questions we investigated. In principle, the capability of computer-based testing 
to branch examinees to easier or more difficult test items, depending upon the 
preceding performance, should increase measurement discrimination. 

Moreover, one would expect conventional measures of reliability to improve for 
at least the following two reasons. First, score variances should be greater due 
to the spread created by branching procedures. Patterson (1962) found this 
outcome of increasing variance to be especially beneficial at higher and lower 
ends of the score distribution in a computer simulation study of sequential 
testing. As Waters (1964) demonstrated in a similar simulation study , sequential 
testing leads to a more rectangular score distribution and potentially a greater 
dispersion of scores. We planned, therefore, to attempt to empirically replicate 



tliis finding. 

As a second contribution to improved reliability , the effect of sequential 
branching should more optimally match examinees^ performance levels with test 
item difficulty levels and, consequently, reduce the amount of guessing behavior. 
Shuford and Massengill (1966) demonstrated theoretically the attenuating effect 
of guessing on test reliability. Thus , we planned to provide for an empirical 
comparison of reUability of a conventional cumulative test and a computer- 



based sequential test in a natural academic setting. 

Since all of the items in a sequential test are not attempted by all 
examinees, conventional techniques for estimating reliability are nonapplicable. 



created and investigated scoring schemes that 
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To circumvent this obstacle , we 
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utilized the hierarchical item difficulty structure of the sequential test and 
allowed for an assignment of all item scores by all examinees. This procedure 
will be more thoroughly covered in our subsequent section on scoring method- 
ology for sequential tests. The important question is whether we can create a 
reasonable and valid procedure lor estimating reliability so that practitioners 
can derive reliability coefficients for sequential tests, especially those under 
construction. 

The implications of sequential, testing for test validity can best be 
viewed within the context cf the '.’attenuation paradox. " As Gulliksen (1945) 
first pointed out, validity and reliability are not monotonically related, and 
undue' increases in item precision will lower the validity of the test. Or , as 
Tucker (1946) demonstrated, medium-ranged item intercorrelations yield the 
better relationship to an ability criterion under conventional scoring procedures. 
Scoring procedures (Gulliksen, 1945), the score distribution (Humphreys, 1956), 
a curvilinear relationship of test scores to ability criterion scores (Brogden, 
1946), and the effects of strata of abilities in the criterion distribution (Lord, 
1955) have been offered as explanations for the "attenuation paradox. " For 
sequential testing, the problem of cumulative correct/incorrect scoring is 
easily avoided. Using various scoring schemes, we hoped to demonstrate that 
a sequentially organized achievement test will rhave ^ higher relationship to a 
criterion ability measure than a conventional achievemept test. This predication 
is based on more optimal branching of examinee’s performance status to levels 
of item difficulty in sequential testing, plus the use of scoring procedures that 
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assign differential 'amounts to item difficulty ieyels» .We also investigated a 
number of ability measures in order to assess empjricaUy the full range of 
possible relationships . 

In regards to eomputer-based sequential testing, the validity of the 
measure niay be improved by the utilization of multiple dependent measures , 
such as confider.ce rating or response latencies, in addition to correct/inccrrect 
responding. In the second experiment to be reported, we investigated the role 
of coFifidence ratings as an additional performance indicator.. 

Since computer-based testing is still. novel in a university setting, 
obvious situational and procedural variables are worthy of investigation. Prior 
experience interacting with the coniP'iter-assisted instructional system employed 
in these studies may provide a f^-cilitating adaptive effect. We, therefore, 
selected for the first e^qjeriment half of the examinees who had at least one 
hour or -more of experience interacting with review and problem materials 
appropriate to an introductory collegiate physics course. Moreover, we were 
interested in determining whether these CAI -experienced examinees would have 
shorter work times as contrasted with the naive e^mine^s. In addition, we 
wished to compare the work time in the computer-testivig situation with 
conventional testing work timess. Krathwohl j( 1959) reported that noncomputer- 
based sequential testing is approximately 2 to 2. 5 times as long as conventional 
cumulative testing. Undoubtedly, this increased test time may be attributed to 
the complex directions and procedures of finding coded sequences in a bulky 
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test booklet. As a related variable, we were interested in the attitudes of the 
examinees towards computer-based testing. 

Scoring Procedures . We turn now to a discussion of- the scoring 
schemes that we investigated in order to both understand their empirical impli- 
cations as well as their impact on the psychometric characteristics of the 
sequential test. V/e utilized the content of a within-term examination in physics 
that covered the topics of mass, force, momentum, energy, and work. For 
each of these concepts , we constructed an item tree network that allo’ved an 
examinee to be routed to difficult or easy items, depending upon his performance. 
Test items according to difficulty were assigned to each of numbered node 

r , 

positions. The initial item had an expected difficulty level of . 50, and each 
subsequent level differed by approximately , 10 of itein difficulty. For scoring 
purposes , each terminal node item had two hypothetical nodes added, in order to 
indicate success or failure on the last item in the sequence. (See items five 
through fifteen in Figure 1. ) The performance of each examinee on' a' concept 
resulted, then, in an ordered array of node numbers. The 16 possible node 
number arrays will be referred to as pathways . 

Insert Figqre 1 about here 

In most sequential test studies, the final node position is assigned a 
ranlt that becomes the examinee^s ^core. In terms of Figure 1, there are five 
final rank scores. We refer to these as Final Node Scores . 
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As there are various pathways that lead to rank final node positions 
hvo, three, and four, different examinees can attempt differing item difficulties 
and still receive the same final score. We tlierefore assigned 16 rank scores to 
the possible pathways, giving the better rank positions according to difficulty 
levels of the attempted items. We refer to these scores as Pathway Ind^ - 
. For purposes of calculating a reliabiUty coefficient, it would be 
desirable to have a score for each of the node posittons and associated test item. 
Given that the test items.are hierarchically arranged by difficulty levels and 
representative samples for that difficulty strata, 've investigated the following 
scheme that makes an assumption about the examinee's performance if he had 
attempted aU items. H an examinee was successful on an item, v;e gave a score 
of hvo for that item, and for all other items below in the vertical array at that 
ncde point in the tree structure. We also assigned a score of one to flie item 
directly above on the basis that there was a probability of .50 of successfully 
.'QTnpmfing it if it had been attempted. If the item was failed, the item score was 
zero for it and all items above in the tree position array. A score of one was 
assigned to the next items below, and a score of two to all other vertical items 
below. In this scoring procedure, the hypothetical terminal nodes were not 
considered. To illustrate, if an exammee failed node item eight, a score of 
zero was assigned to node items four and eight, a score of one to node i 
eleven,- and a score of two to node item thirteen. If the examinee had passed 
node item eight, then he would be given.a score of two on node items eight, 
eleven, and thirteen, and a score of one on item four. Tiiis scoring procedure 
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makes a strong assumption about the monotonic relationships of item difficulty 
and performance. We refer to this scheme as the All-Item Sco^. 

For purposes of comparison, we al^o considered a^conventional 
cumulative score for attempted items. This is referred to as the SumScore. 
Thus, we considered four scoring schemes for each of the five subtests-. A 
total test score for each scheme was a sum of the five subtests. 

Materials. The computer-based sequential test was designed to be a 
parallel measure of achievement found in a one-hour classroom examination 
that covered the topics of mass, force, momentum, energy, and work. Items 
for the sequential test were selected from a large pool of items collected from 
exams given by the same professor in previous years. The item difficulty 
statistics were somewhat ambiguous in that they represented many different 
student populations and were found in both within-term and final examinations. 

To better insure the item^difficultyievels, a panel composed of the professor, 
two physicists , and two physics curriculum writers rated all the potential items 
into seven levels of difficulty. For the selected items , all of the panel agreed as 
to the level of difficulty. ^For the topics of mass, momentum, and work, our 
inability to complete the seven levels of difficulty led us to restrict the size of 
the item tree to three arrays or six items representing five levels of difficulty. 
We were able to achieve the seven levels of difficulty for the topics of force and 

energy. 

Since an examinee attempted only a subset of items due to the pass/fail 
decision rule that directed the student to the next higher or lower node item, the 
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total test consisted of 17 items from a total of 38 available items. AU of the 
test items as well.as the 20-item classroom test were in a multiple-choice 
format. Figure 2,illustrates a typical item plus the confidence raKng format 
utiUzed in Experiment Two. The directions for the confidence rating were ^rt 
of the general intepduction to the testing situation and required the examinee to 
indicate his confidencje over a nine-point scale. 

Insert Figure 2 about here 



rjAneral Procedures . Since some students were unfaimliar with our 
CAl system, we briefly explained to all participants the procedures for activating 
the computer program, how to enter an a^wer, how to make corrections, and 
how to sign off OJT .terminate. This inbroduotion took less than five minutes due 
to the simplicity of the procedures; no additional help was reqatea. Scratch 
paper was provided for calculations, and answers typically consisted of the 
numbers "I" through "5" plus an entering response. The G.M system response 
typically was two seconds before 8ie next item was presented. 

The test was controlled by an IBM 1440 CAT system, and the actual 

presentation was given via IBM lOSO terminals. There are seven of these 

typewriter, terminals at the FSU-CAI Center; each terminal is located in a 
separate, isolated room. Each student proceeded individually, and toW work 
times were collected. The computer-based test was administered approximately 
one week after the regular class examination. It required three days to schedule 
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8.nd process all of the participants. 
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Experiment I . The first experiment, in essence, was an initial study 
in the empirical feasibility of computer-based testing. The ^estions relating 
to reliabiUty, validity, and procedural factors were of primary interest. Fifty- 
six freshmen (25 men and 31 women) were randomly select^ from approximately 
480 students enrolled in the Fundamentals of Physics course in the winter term 
of 1967 at Florida State University. Half of the selected participants had prior 
experience with the computer terminal at the CAI Center. This prior experience 
consisted of review physics problems for the preceding class examination. Each 
had taken a 20-item class examination the previous week. 

Results . In order to evaluate the appropriateness of the item difficulty 
assignment, the fact that all examinees attempted the first item on each of the 

five subtests' offers evidence as to the accuracy of the test construct procedures. 

\ 

As indicated in Table 1, the empirical difficulty level approximates the desired 
level of . 50. 



Insert Table 1 about here 

In regards to the interrelationship among the various scoring methods, 

it can be shown that there is a monotonic constraint as to the overall score for 

1 

the four methods; that is , a simple sum of the correct items will be directly 
related to the final node position.' As indicated in Table 2, the intercorrelations 
of total scores among these various scoring methods are substantially high. It 
is worth noting in Table 2 that the hierarchical All-Item Scoring scheme has the 
lowest relationship to the other procedures. StiU though, the relationships are 
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close enough so as to encourage the use of the All-Item Scoring procedure for 
the caloulatton of reliabiUty estimates. Given these high relationships, and 
for the purposes of simplicity presentati<m, we shaU use the Final Node score 
and the hierarchical All-Item score for the determination of otter psychometric 
charsicteristics of th6 sequential test. 

Insert Table 2 about here 

Since one of the objectives of the sequential test is to achieve a uniform 
score distribution, we tested the Final Node scores and the hierarchical AU-Item 
scores using the Kolmogorov-Smimov cumuUtive distribution goodness-of-fit 

test. In both cases, there was no significant difierence from the uniform 

' ' * - ' • * ♦ • 

distribution (P>.30) We agmn interpret this outcome as evidence that the 
empirical outcome for the All-Item scoring procedures is maximizii^ the spread 

in the score distribution. 

ReiteMUty. As has been noted above, only the hierarchical All-Item 
scoring procedure aUows for the use of conventional reliability estimation 
techniques. As can be noted in Table 3, an analysis of variance techmque 
(Raijinowitz & Eikeland, 1964) approach to estimating a Kuder-Richardson 20 

for stratified tests indicated that the overall reliability for the sequential 

? * 

physics test yielded a coefficient of . 885. Perhaps even more encouraging was 
the high subscale reliaMlity coefficients. The higher estimated coefficients for 
force and energy can be attributed to the fact that these had more items present 




in the testing sequence. 
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Insert Table 3 about here 



In comparing the sequential test with the conventional test, it was 
found that the 20-item classroom test yielded an overall K-R 20 coefficient of 
. 515 1-he subscale clusters tended to range from . 141 through . 398. In order 
to provide a fairer comparison due to differential length, one can utilize the 
Spearman-Brown formula to increase the 20-item conventional classroom test 
to a «=iTT.i1ar 38-item pool employed in the sequential test; this yields an estimated 

** a' * • , - - 

reliability of . 668. In boCi cases , the reliability coefficient for the sequential 
test is significanUy greater than that for the 20-item classroom test. Moreover, 
the reliability for the sequential test compares quite fevorably with that found for 
most standard achievement tests currently available. We also feel that the 
evidence’from the relationship of the liierarchical All-Item scoring procedure to 
the other scoring procedures offers some sustaining evidence that this approach 
for estimating reliability for a sequential test is viable and sound. We would 



contend that a procedure to calculate conventional reUabiUty esti.^tes will be 
invaluable to investtgators who are constantly reworking tests for investigatory 



reasons. 

Validity . The method which was chosen for validaUng the computer- 
based sequential testing approach and the accompanying scoring schemes was 
the correlation of these scores with the score on the conventional class test plus 

I-'-;. . 

the score on the final classroom examination. These are presented in Table 4. 
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Insert Table 3 about here 

1 «/\n n/^pfficieuts are statistically significant, 
Altliough all of the resulnng correlahon coeSicxeuts 

« »«.».» >«»«" 

P„MP, «. -)» ~~ >» 

„»« 1 «» cl» «■ “ 

1 SiB Moreover, the mean score for the 20 item 
20 reliability estimate of only . 

s v,ith a standard deviation of 2.3. This skewed 
classroom test v»as 15. 5, with a sianoa 

.ntr^ution may have reduced the correction coefficients due to a restricUon 
. ^e in the scores: It is also worth noting that the rela« between the 

1 1 .the final -rade in the course is substantially lower than fliat 

classroom test and the final graue 

I., th. 

rfip pnlof 49 with the final grade. (See Table 5.) 

score yielded a correlation coefficient of . 49 

Insert Table 5 about here 

« , 1 .1 was a better predictor of the final achievement of the 
Thus , the sequential test was a be P 



students . 



iPeAaps the most persuasive 



substantiation of the worth of the sequential 



testing Wonch is indicated in the more 



substantial correlation between the 



hierarchical All-Item scoring outcomes 



and that of the participants’ score on the 



Florida Twelfth Grade College Lxamination 



The Florida TweHth Grade College 




Sx2.inixi3.tioii is 2 l sum of a verbal and mathematical aptitude score. As is 
indicated in Table 5, the hierarchical All-Item correlation coefficient with this 
ability measure is .43. The relationship to the conventional class test is 
substantially lower or is statisticall3»^ nonsignificant from zero. As indicated 
in Table 5, the sequential test scoring procedures jdelded moderately better 
relationships to the ability measure than did the final course grade. We feel 
that this first sequential testing experiment has documented the potential efficacy 
of this approach to improvii^ the relationship of collegiate achievement to ability 

selection criteria. 

TestinprTime and Administrative Factors . An issue of prime importance 
is the administration and adaptation factors required by computer-based testing. 
As indicated in the design, there were two groups which varied as to prior 
exoeriehce with computer interactions. In terms of their test performance, 
there ^re ho significant differences between the group means (P >. 15). There- 
fore, the familiarity with the computer terminal and its operation appears to 
have no discernible effect on the examinees’ scores. As noted in the procedures 
section, the introduction to the computer terminal operation is brief and simple 
in nature. Computer respohse time also allows for a smooth flow of questions. 
Thus, each examinee has opportunity for self -pacing through the test materials. 

The time used by the students in this study was carefully recorded under 
both testing conditions. Utilizing a two-way analysis of variance with prior 
computer experience arid wrirk time as the factors , we found a sigmficant 
difference in favor of computer-based testing in terms of work time (P 05). 
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The mean time for the 17 items on the computer system was 25. 04 minutes as 
compared to 30.73 minutes on the 20-item classroom test. These mean times 
average out to 1.47 minuies per item under computer control in con^rison 
with 1. 54 minutes for the classroom test items. Again, there was no significant 
difference with regards to prior computer-interactive experience nor a significant 
interactive effect. The correlation between the time required for the two test 
situations was r = .21. We interpret this low relationship as indicating consid- 
erable individual differences in work strategies in these two testing situations. 
Further investigations should explore the cognitive and/ or personality variables 
that would predict work time in varying testing situations. 

It should be noted that the computer-based sequential test was more 
difficult in nature and, consequently, in principle should have slowed the 
examinees* work rate. Moreover, each of the questions was typed out at the rate 
of 13 characters per second.*- This type-out rate also reduces the amount of time 
in comparison to a conventionally presented test. As noted in the introduction, 
the computer-based work times are exceedingly favorable in comparison to the 
times for convenfional sequential testing (at least three to four times longer) as 
reported by Krathwohl (1959). Thus, the improvements in computer technology 
have decreased the overall test-taking time for the examinees in sequential 
testing situations. 

We view the outcomes of Experiment One as establishing the feasibility 
of computer-based sequential testing. Moreover, we find the investigation as to 
various scoring procedures of value in that we were able to generate a seemingly 
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appropriate scheme by which one can provide for conventional item reliabilily 
estimation procedures. . Moreover, both the reUabiUty and validity cl^cter- 
istics of the sequential test were superior to that of the conventional classroom 
test. Perhaps the most noteworthy result was the better , but moderate, 
relationship between the sequential test and the ability criterion measure, 
fi nding direcfly relates to the discussion on the "attenuation paradox" and indicates 
that sequential testing may provide a viable way of increasing item precision 
without introducing a ceiling effect on validity relationships. 

Experiment II . The second experiment focused on the relationship of 
examinee’s confidence ratings on completed test iteins to the other variables 
relating to reUabUity and vaUdity. More explicitly, we wondered if the prediction 
of concurrent achievement such as the class test, the final course grade, and the 
ability criterion measure can be substantiaUy jmproyed if confi4ence rating scores 
are combined with the sequential test score. In addition, we ^thered attitude 
reactions on the part of the participants towards the computer testing situctton m 
order to assess any negative reactions. Obviously, we also wished to replicate 
the findings of the first experiment an that the All-Item scoring procedure led to 

■improved le’iiability and validity relationships. 

Thirty freshmen were randomly selected for this study from approxi- 
mately 300 students enrolled in the physics course at FSU during the spring term 
of 1967 . AU of the subjects were naive in terms of their exposure to computer 
interaction. Each examinee had taken a newly constructed 20-item class test 
approximately one week prior to the computer-based sequential test. 
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In order to effectively §sli ti confidence ratings on test items , introduc 
tory directions were presented as portrayed in figure 3. In addition to the 
explanation^ one sample problem with an opportunity to rating one’s confidence 
was included in the pretest int’. eduction. Thus, each examinee had an oppor- 
tunity to solve a very ^-imple physics problem and give a confidence rating. 

This simple introduction to confidence rating was pretested with college students 
here at FSU and has provin to length^^'^ the introduction by approximately three 

minutes . ' 

Insert Figure 3 about here 

Results . AW in the prior examination, the mean correct proportions on 
the initial item of the five s'ubtests were bounded between .43 and . 57. This lends 
further evidence' that tll4 item difficulty levels of the sequential test were 
appropriately assigned, Ifib interrelationships between the four scoring methods 
and the' confidence scores are presented in Table 6. 

Insert Table 8 about here 

I ’ 

The interrelationships among the scoring methods are almost equivialent to those 
found in Experiment One, in that the high intercorrelations among the scoring 
procedures are still present. The relationships with the confidence ratings , on 
the other band, were considerably lower although comparable to similar results 
found in learning studies for confidence ratings. 
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‘ 'in terms*o^ reliability,’ the hierarbhical All-Item scoring approach 
yielded a more substantial Kuder-Richardson 20 coefficient of . 904; the subscale 
reliabilities were bounded between . 58 and .88. In comparison, the classrooin 
test yielded a K-R 20 reliability of .69^! Again, the computer-based sequential 
test appears to have'better reliability characteristics. The K-S goodness -df-fit 
test for a uniform distribution ihiiicated 'no si^ficant differences. These' results 
further substantiate the reliability outcomes of Experiment One. 

As can be seen in Table 7, the validity relationships tend to replicate 
those found in Experiment One. The class test has improved as a predictor of 
the final grade performance. The confidence ratios are irioderately related to 
the validity measures. An analysis of variance approach to miiiti^le regressiori*’ 
was performed in order to determine the conibined relationship of the sequential 
test scores plus the confidence rating. Regressing ori the parall^r class test, the 
combined nieasures of the All-Item score plus the confidence sco^e yielded a 
multiple R of \'4^8. The multiple R improved for the final bourse grade to .739. 



Insert Table 7 about here 



An improved prediction of the Florida Twelfth Grade ability measure yielded a 
multiple R of .618. In all cases, the beta weights of the hierarbhiiidi All-Item’ 
sequential score plus }.he coididence score were significant, althdugh tbe All-Item 
score had substantially higher beta, weights .' ' InTerms of this experiment, the 
availability of a confidence rating score does improve the prediction of both 
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concurrent achievement and ability criterion measures. Obviously, computer- 
based testing allows for the gaining of these multiple dependent measures with 

relative ease. 

Turning now to the analysis of work times for the class test and 
computer test, there was no statistical difference. The mean time was just 
over 30 minutes for total presentation; the item time for the 17 CAI items plus 
confidence rating was 1.79 seconds, and that for the con/eational class was 1.71 
seconds. Thus, the time fector was equivalent for both testing situations. 

At the termination, the examinees filled out an atti'-idinal scale adapted 

I - 

from a CAI Attitudinal Scale prej^r«d by. Brown and Gilman (1967). Table 8 

- • • i . * 

presents the ten item statements that were rated on a five-point scale tliat _ 
typically ranged from "strongly disagree" through "uncertain" to "strongly 

* f 

agree. " In Table 8, the mean value and nearest associated word from the scale 
are also presented. Balancing for positively and negatively worded statements , 
the examinees reacted to the computer-based testing situation with favorable 
ratings . Item 4 is especially interesting in that the vast majority of examinees 
reported that they never guessed at ansvvers. We interpret this report as 
indicating sequential testing minimizes guessing since item difficulty is being 
adjusted to prior performance. These positive attitudinal results are similar to 
those found for -a well -prepared CAI course presentation. 

Insert Table 8 about here' 
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Cost Factors. The current and future costs of computer-based testing 
wiU vary considerably due to a number of fectors as foUows: (1) the kind and 
cost of the computer terminal. (2) the size of the computer system, (3) the use 
and cost of teletransmission,' ar:d (4) the amount of real-time analysis required 
to utilize multiple dependent measures. Cur current hourly cost at FSU.for. 

IBM 1440 terminal time is $2. 56. This is obviously many orders of magnitude 

higher than the costs for conventional testing. .. . 

Two additional factors may also increase these cost estimates. First, 

sequential testing requires two to three times the number of test items, depending 
on the size of the item tree structures. Consequently, sequential testijig has a 
high., cost where new item construction is required. Secondly, the tqst items 
must be encoded for the co iputer system. This typically costs approximately 

$. 50 per item. 

There are many possible savings that accrue to computer-based test^. 
Obviously, the costs of printing, handling, etc. , are included in the costs of the 
terminal charge. The costs of scoring and processing answer sheets at a test 
center are also included in our charge for terminal time. Thus , many of the 
hidden lo^stic costs of conventional testing are minimized within the computer 
approach. In the future, one can anticipate further cost reductions in. computer 
applications, whereas it is difficult to anticipate any cost savings via qonventional 
approaches. Ultimately, any improved reliability and validity characteristics of 
computer-based sequential testing will have to be compared in a cost/utility 
sense against the increased fiscal costs of this technological approach. , 
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Summary. Our empirical study of computer-based sequential testing 
substantiates for us the feasibiUty and potential worth of this methodological 
approach. Cur investigation of various scoring schemes indicates the close 
empirical similarity of the various procedures. Moreover, the technique for 
occigning values to all items via the hierarchical AU-ltem scoring procedure 
appears to resolve the problem of estimating reliability within sequenttal testing. 
The replicated evidence on the high relationship among the scoring procedures, 
the performance on the common initial item in each of the five subtests, and the 
score distributipn analysis support our view that the All-Item souring procedure 
can be uttlized without undue jeopardy by an investigation in estimating test 

reliability for tests under development. 

More importantly, the relationship between confidence ratings and item 

scores plus the consequent improvement in multiple S prediction of concurrent 
achievement or related ability measures represent the strongest empirical 
evidence, from our viewpoint, as to the potential worth of computer-based 
sequential testing. Future investigators may wish to explore the use of 
confidence rattiigs or other related dependent measures in their techniques for 
assigning item scores in order to generate better reliabili^ estimates. The 
improvement in the relationship behveen computer -based sequential achievement 
. testing and validity has direct implications for college selection procedures. If 
the relationship between the college selection tests and college academic 
achievement can be improved via sequential achievement testing, then many 
obvious cost and human savings can be effected. 
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The results on v^ork time indicate to us the desirability of utilizing a 
computer technological approach to sequential testing. While the costs of student 
testing time can be assigned a variety of values, we would contend that shortening 
test time aUows for a more broadly based approach to academic assessment. 
While the attitudinal findings are of a very preliminary nature, we do take 
encouragement in that no negative fectors were encountered in our studies. 

In terms of the fiiture, there are obvious needs to further explore the 
size and shape of the item tree structures as these relate to theoretical consider- 
ations within test theory. While our approach has been empirical in nature, we 
could claim that further exploration as to the theoretical nature of sequences of 
item structures will be required before a broad understanding of this approach 
can be achieved. Moreover, the obvious information retained in response 
latencies has not even been explored within these experiments. There are 
sufficient findings from learning experiments which indicate that response 
latency may prove to have just as powerful an impact on the predictive relation- 



ship to criterion measures as did the confidence ratings. 

As a last note for future study, we would surest that computer-based 
approaches to testing may allow for an acceptable and feasible way of controlling 
test anxiety. The conception is to adjust the item difficulty level for each 
examinee in order to minimize the extreme anxiety reactions found when 
examinees are working on impossibly difficult test items. We have demonstrated 
in a learning experiment (O'Neil, Spielberger, and Hansen, 1968) that state 



rs-tcs sire highly related, and that 
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anxiety ratings, blood pressure, and error 



so 



anxiety status can be manipulated according to the difficulty level of the learning 
materials. The future of computer-based sequential testing, therefore, *ias 
many avenues of needed investigation. We trust this papei will provide a new 
stimulus for further research in the area. 




FIGURE 1 



CONCEPT ITEM TREE NETWORK, PATHWAY 
STRUCTURE AND NODE NUMBERING 
FOR SCORING METHODS 





Fiiml Node Position 
Rank Index 

1 



2 



3 



4 
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FIGURE 2 



TYPICAL TEST ITEM FORMAT 




A certain body is observed to be moving with a constant momentum 
of 100 kilogram meters per second. Ten seconds later the same body with the 
same mass is observed to be moving with a constant momentum of 200 kilogram 



meters per second. We can conclude that 

1. a net force acted on the body during the ten seconds we weren’t 

observing it. 

2. no force acted on the body during the ten seconds. 

3. this change in momentum is not associated with an acceleration 

of the body. 

4. the velocity of the body has not changed. 

5. none of these. 

(Answer) 

Confidence 



FIGURED 



DIRECTION AND FORMAT FOR THE 
CONFIDENCE RATING 

* 

After" each question of this test you will be asked to express the 
confidence you place in the correctness of your answer. This expression of 
confidence will be based on a scale from 1 (which signifies no confidence) to 9 
(which signifies extreme confidence). 

The following scale will be helpful to you in rating your confidence: 

no confidence 1 

moderately unconfident .... 2 



mildly unconfident 3 

slightly unconfident 4 

uncertain ; . . . 5 . ■ 

slightly confident ..... . 6 



mildly confident 7 

moderately confident 8 

extremely confident 9 

Following your response to each question, ’’confidence will be 

typed at which time you are to reply with an integer from 1 to 9 in the blank 
provided. When you need to refer to the scale at any time thereafter use the 
mimeographed sheet next to the terminal. 
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FIGURE 3. — Continued. 



SAMPLE 



Now answer the following sample question and, when indicated, rate 
your confidence in the correctaess of your answer. 

E^qjresc the following number in scientific notation: 502,785. 



1. 502785 X 10**5 

2. 5.02785x 10**5 

3. 5.02785 x 10**6 

4. .502785x10**7 

5. 502,785 X i0**2 

. (Answer) 2 
Confidence 

You have rated your confidence between 6 and 9 which indicates that 
you have answered the question with some degree of certainty. The correct answer 
is 2 (5. 02785 X 10**5). 




TABLE 1 




table 2 ■ 



PRODUCT-MOMENT CORREl^TlONS' AMONG THE 
FOUR SCORING PROCEDUiRES ON THE 
SEQUENTIAL PHYSCS TEST ' 




(1) Final Node 






• • ; - 


(2) Pathway Index 


- - ■ .912 






(3) All-Item 


- . 887* 


- .904 




(4) Sum Score 


- .936 


- .906 


.843 



^Negative correlations are due to the rarJting procedure. 
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table 3 



CONCEPT SUBSCALE • 

KUDER-EICHAEDSON 20 HELIAHLITY ESTIMATES 



Concepts 



Mass Force Momentum Energy Work Total 



Reliability 

Coefficient 



,681 .847 



.718 



829 



736 .885 






87 



o 

ERIC 



table 4 

PRODUCT-MOMENT CORRELATION BETWEEN THE 
SEQUENTIAL SCORING PROCEDURES, THE . 
--iciAS^OOM TEST, AND THE 
■ final GRADE 



Scoring Methods 


Classroom Test 


Final Grade 


Final Node 


- .32 


- .41 


Pathway Index 


- .28 


- .43 


All-Item 


.32 


.49 


Sum Score 


.39 


.38 



TABLE 5 



INTERREIATIONSfflPS OF SEQUENTIAL TEST SCORES 
AND VALIDITY CRITERION MEASURES 



— - 


1 


2 


3 


4 


(1) Final Node 










(2) All-Item 


- .89 








(3) Class Test 


- .32 


.32 






(4) Final Grade 


■- . - *41 


.49 


.19 




(5) Ability Measure 


-.37 


.43 


.13 


.34 
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TABLE 6 



INTERCOBBEIATIONS OF THE SEQUENTIAL T]EST 
SCORING^ PROCEDURES FOR 
“EXPERIMENT TW'GP ' -- 







1 


2 


3 


4 


(1) 


Final Node 










(2) 


Pathway Index 


..907 


- . 






(3) 


All-Item 


- .893 


-.898 






(4) 


Sum' Score ' • 


- .923 


- .904 


.862 




(5) 


Confidence Score 


- .271 


- .283 ■ 


.389 


.192 



TABLE 7 



INTERCORRELATIONS OF SEQUENTIAL TEST SCORES, 

THE ACHIEVEMENT MEASURES, AND ABILITY 

MEASURE FOR EXPERIMENT TWO 





i 


2 


3 


4 


(1) All-Item 










(2) Confidence 


.39 






- 


(3) Class' Test 


.34 


.18 






(4) Final Grade 


.56 


.43 


.31 




(5) Ability 


.49 


.37 


.11 


.32 



TABLE 8 



ATTiTUDn'LAL RESPOa^SES TOWARIDS* COMPUTER-BASED 

SECUERilAL TESTING 





^ 

- z: i : “ ' : * 


Mean 

Scale 

Value 


Nearest 
Associated 
- Word 


1. 


While-taking ti‘3 computer -test, I felt- 
challenged to do my best. 


4.1 


A 

Agree 


2. 


I was concerned that I might not 
understand the material. 


3.9 


Agree 


3 . 


Wiiile taking the computer test, I felt 
isolated and alone. 

“ i 

• • - 


2.6 


Some of 
the time 


4 . 


I guessed at the answers to questions. 


1.2 


Very 

Seldom 


5 . 


I was more involved in running the 
machine than in understanding the 
question. 


1.1 


Never 


6. 


I was aware of efforts to suit the 
material specifically to me. 


2.4 


Disagree 


7 . 


The computer situation made me 
feel quite tense. 


2.3 


Disagree 


8. 


Questions were asked which I felt 
were not relevant. 


1.0 


Never 


9 . 


I could have done better 2 'f I hadn’t 
felt pushed. 


1.3 


Strongly 

Disagree 


10. 


I would say computer testing is . » 
superior to class testing. 


4.2 


Agree 
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learning outcomes of a computer-based, 

MULTIMEDIA INTRODUCTORY PHYSICS COURSE^ 

Duncan N. Hansen 
The Florida State University 

This report on FSU*s computer-based course in introductory 
physics, hopefully, mil contribute to the growing- evidence on the 
efficacy, limitations, and requirements for a technplp^ical approach 
to instruction. The existence of regular computerrbased, acpredxted 
courses precedes the FSU ^oject in. at least two ipstancq^.. The 
Stanford Project is in the second year of autonomoqs.^ -computer-based 
tutorial instruction in reading .and mathematics, .for prxmary.^grade 
children. The University of Illinois' Project has offered a qc^plete, 
accredited course in library science for graduate students. .%xle many 
other partial CAI course applications could be cited, the new contrx- 
butions of the FSU Riysics ^oject can be viewed in terms of four 
salient features. 

First, this introductory physics course for non-science 
majors offers the opportunity to study .a curriculim that has a very 
rapid increase in conceptual complexity and inter-yelatedness . As wxll 
be illustrated in the learning results to be presented, the typical 
learner in the conventional course has a steadily decreasing performance 

1 Paper read at Americar. Educational Research Association, 
Chicago, February, 1968. 
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curve for this curriculum. Consequently, the CAI physics course has 



ample opportunity to improve learning outcomes in addition to estab- 
lishing the feasibility of a technological approach to instruction. 

Focusing on a beginning collegiate level course allows 
one to study' £he feactions of the fastest growing ^student populations 
in the United States. If the costs of CAI courses are to be justified, 

<. rapidly expanding student population group offers an excellent crucible 
in which to document the benefits of the technological approach. Thus, 
the' study of collegiate freshmen is a new area of. exploration for a 

CAI project. .• . . 

Ait 'CAI covirses stress -the feature of ‘ individualization. 

The FSU physics ‘bourse manifests this goal in terms .of .-a number of 
student-controlled decisions. For example, the FSU . students * controlled 
" Wrass scheduling, the opportunity to repeat presentations, and the 

amount and kitid of feviw materials. Thus, these freshmen had a unique 
" opportunity to control and regulate thfiir learning experiences. This 
style of individualization is uniqudj in degree at least, for CAI 
tutorial courses. " 

Finally, the FSU physics course utilized a wide variety- of 
multi-media approaches. Rather than restricting- the informational 
' presentation to the terminal devices of the 1500 computer system, we 
chose to employ as many media alternatives as possible. This feature of 
the physics cour^b offers us an opportunity to analyze the impact of 
these mekih types on contiitr^t concept acquisition behaviors as well 
’ as oh the final exam. Thus, the physics course may provide information 
about the relative efficacy of different media approaches. 




This report will stress the developmental factors present 
in creating this computer-based physics course as well as the empirical 
outcomes of the comparison with' the conventional lecture apporach, We 
report our solutions to the problems of selecting the participating 
students, the logistics of implementing the course, and the .preliminary 
manpower-cost considerations so that othe-f ifivestigators may assess the 
full requirements for this kind of educational resfearchir. 

n»c»arrh Strategies . The creatioh'-iha- evaluation. of .a 
technologically based course of instruction at- the coll egiate .level 
are fraught with many developmental and methodologi'cai problems. 

During the conception and early phases of this project , a considerable 
effort was devoted to providing a reasonable developmental design 
and the appropriate collection of data upon which one could answer 
questions generic to the "behavioral "Engineering" or "systems approach" 
to course development. To be' inor'e specifih, one must be able to 
specify in precise terms the need for additional revision based upon 
fruitful inferences derived from an empirical data base as the curricu 
lum is undergoing the developmental process. -Our stcategy .feo,.5esolvxng 
this and related formative, evaluations! problems. can be viewed, in a 
series of systematic procedures moving from the most general to the 

most specific*. 

At the most molar level, professional educators are con- 
cerned about the overall outcomes of a course. These .. outcomes are 
typically Judged in terms of performance oh niid-term. and final, examin- 
nations. Commonly found in most colleges, and at Florida State University, 
is the requirement for common examination proceuures for the assignment 
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of course -grades* To improve the precision of this comparison, we 
planned on utilizing three groups, for the basis of general course 
" cbmpatisons.'-'The first group, the CAI condition, consisted of those 
students- •taking all of the autonomous, multi-media instruction at the 

i ’ 

CAf tetiter. The GAI group’s only participation in conventional activities 
was to take the paper-.and-pencil, mid-term and final exam. Since the 
CAI students were self paced in nature, the raid-term and final were 
offered on a flexible basis when the student reached the appropriate 
point in the course, 

- ■' ' Since we are aware in educational research of the confounding 

influence of new instructional ^ctiyities falling under the rubric of 

th4' "Hakhorhe effect", -.it is. difficult to make a valid cross-group 

ev^hluative statement. To overcome this obstacle, a large number of 

»• 

the students attending .tdie. Physics 107 class were offered limited CAI 
interaction 'ebver.ing review, homework, and test topics. We contend 
that the sfeiidents in this partial CAI group, which had two to six hours 
of terminal experience, provide a valid way of assessing the general 
■'"influence of a new, exciting approach to instruction. Obviously, we 
' planned for a matched group of nonrCAI experienced physics students, 

•■' ' - As an additional advantage at a more precise level, the 

partial CAI group provides data on classes of problems parallel in 
nature to those found within the technologically based autonomous course, 
diiir strategy -was to establish a baseline by xdiich we could judge whether 
various curricular changes were really providing for improvement. The 
review and pro'blem ses.sions offered to physics students prior to 
■'three within-term^ and;rfinal examinations provided a natural setting 



in which to collect data in order to judge performance on the concepts 
covered in the course. As will be explained in the phase development 
of the project, the collection of this data was replicated over three 
terms so that we feel we have stable estimates of concept difficulty 

for undergraduate students at Florida State University. Moreover, we 

■> « , 

'• - ■ 

feel that comparing CAI students ‘ performance to baseline data on the 
physics concepts provides a more precise evaluation of the impact and 
efficacy of the technologically presented concepts. This comparison 
helps estimate the need for and degree of required revision. 

At the response analysis level of evaluation, the CAI 
approach to course development provides complete records of responses 
and latencies as collected in the nprmal course on instruction. We 
have devoted a .great . deal of effort to the development of a data 
management ^system, by. ;idiich. these responses can be appropriately 
sorted intq^.meaningful groups for inference-making about the substance 
of cpur.se .reyisions.. Our curriculum writers are finding this evidence 
to be . ex.ceedingly useful in the rewiting of specific presentations in 
ordep to., accomplish the behavioral objectives set for the course 
concepts. The judgment about the attainment of behavioral objectives 
is a difficult one at best and, we contend, best judged in terms of 
concurrent learning performance as opposed to the sample of behavior on 
the final examination. 

At the most detailed level of research, we are starting 
to build descriptive models that would characterize the learning 
activities of the students. The aim of these models is to provide, first. 
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a detailed description of the students' learning activities over the 
topics found in the course. Hopefully, individual parameters can be 
estimated within these quantitative models in order to provide empirically 
based decisions concerning the need for additional remedial or enrich- 
ment -instruction for the student. It is within t^ese berms that we 
are hoping to provide a major step forward in te^ms of individualizing 

instruction within„this physics course. 

Developmental Plans . To implement the project strategies 

and to insure a meeting of our time table for field testing, we 
developed a phased set of plans. As general characteristics of these 
.. pi;ins were two organizational features, namely, the require ment for 
role differentiation of faculty and project staff and the assignment 
of tasks to appropriate developmental stages in order to provide for 
sufficient review that allows for. appropriate revisions where required. 

In considering a feasible, and perhaps optimal, way to develop a 
technologically based course within the structure of an ongoing 
University program, it was decided to develop first those parts of the 
course which could be field tested and evaluated as to their validity 
. and. contribution to the course objectives. Given this guiding criteria, 

the following phases of development took place. 

Hiase I consisted of the development of a detailed topic 
outline for the course of instruction. The involved physics professors 
gave innumerable hours in describing the nature of their introductory 
course for non-science majors and the techniques by which they presented 
. concepts such as demonstrations, films, etc. This led tp a tentative 
course content outline of a reasonable degree of specificity sufficient 
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so that appropriate review and problem material could be written for 
each of the topics. These review sections were offered on the CAI 
system during the fall term of 1966. The material represented a 
parsimonious statement of the essential concepts found within the 
topic areas as well as an opportunity to work on problems and multiple- 
choice test items parallel in content and form to those to=be found on 
the examination. In nosi cases, the students ,«re required to identify 
important definitions and relational statements, solve problems, and 
apply their formed conceptions to test items. Thus, the materi 
could be characterized as an alalogous representation of homework 
assignments and test content given under ' CAI presentation. This 
activity has grown to be exceedingly popular at FSU. He now process 
nearly 300 students a term mth this material. This phase was completed 

by Christmas of 1966. 

Phase II consisted of the detailed response an^alysis and 
revision of these review materials. This re^^sion provided us with 
some indication of how close we were coming to providing problem 
material which effectively changed the students' behavior in terms 
of subsequent presentations within the program as well as performance 
on concurrent class examinations. In addition, we video recorded all 
: 0 f the lecture presentations offered in the conventional classes. 

These were analyzed for conten^ and for level of specificity. Then, 
a tentative statement of course behavioral objectives by concepts was 
generated. The first analysis of the assignment of media to concept 
presentation was made at this stage. As an additional activity for 
this phase, all of the film and demonstration materials for the 
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conventional course were progra^d from a problem-review point of 
view. These were included in our review sessions offered to students 
from the coiventional class in the winter and spring term. This phase 

lasted approximstely four, months . 

In Phase III. a final version of the course bdiavioral 

■bbiectives was determined. Figure .1 presents a typical example of 
the Sheets utilized by the project staff. Tnese underwei^t many revisions 
and a more specific level was developed for the detailed presentation 
of'ikterial. ^e results from the video content analysis were then 
utilized in the final assignment of media presentations. The primary 
activity of the third phase consisted of the constciccion of the total 
CAI course. It is important for you to understand that the course is 
multi-media in nature.and only uses CAI interaction where deemed 
appropriate after sufficient analysis of the content and behavioral 



objectives , 



Insert Figure 1 about here 

Thus, the course consisted of series of text material, film loops, 
audio presentations." conventional filz. presentations, and extensive 

CAI iteration. Figure 2 presents a course flow diagram for the topic 
of light and optics presented in Figure 1. As you can see. we attempted 
to build lesson structures that would take approximately one hour in 
duration. The course was completed in early September of 1967 for 

studBiit piT6S6iitatioii« 



Insert Figure 2 about here 
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Ihase IV. being the first full-trial test. »as completed in 
December of 1967. He are presently in Ihase V which is an analysis of 
the field test outcomes for the purpose of revising and extending the 
materials in ordef to better accomplish our overall goal of individual 
izing and maximizing the learning of frehhman students in an .introductory 
physics course. The revised course will bi field tested again: in the 

spr:.ng term of this acadeniic year.- 

and Student Sel ection Procedure. The granting 

of accreditation to a totally new instructional approach is fraught 
with many obstacles. Foremost, the universitg, administrator wishes 
assurance that- the =T.ew course" will be equivalent- to or better than 

the existing course. Prior to the firSt field test, one must appeal . 
to a -host of tangential arguments kno^hg that empirical outcomes 
might contradict these claims. At FSU. the evidence from the develop- 
mental physics CAT review activities indicated a gain of ten to 
twenty percent in conventional examination performance; This evidence 
was most persuasive. Secondly, the conventional course instructor, 
professor Steve Edwards, has been an active mead>er of the project staff. 
The positive support and recos»iendation of Dr. Edwards was undoubtedly 
the critical factor in gaining acceptance for the first field. trial. 
Even though, the following restrictions became part of our operating 



policy: 



1 . 



2. 



The course instructor will retain final editing control 
of all the CAI phy-‘?ics materials. 

If. in the judgment of the course instructor, this study 
is proving :to be disadvant geous to the students, he will 
terminate the experiment at that point. 
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3. 



A qualified physics proctor will be available during 

all instructional sessions. 

The CAI students consulted these physics proctors on 
four occasions during the course. Thus, the need for off line 
physics proctors to help solve conceptual problems seems unwarranted. 
Iftidoubtedly, the extensive writing and editorial help of Professor 
Edwards plus the content analysis of the video recording of the 
conventional course insured that the conceptual coverage was sufficient 

and styled in an appropriate, meaningful way. 

.As a final factor in acceptance, the course grades for 

the CAI course were to be derived from their performance on the 
conventional course examinations. Since we msh to compare final 
performance among the three groups, this was a desired condition for 
the field study. Presently, we are analyzing mthin course performance 
in order to find more reliable indices that might replace the exam 
requirement . 

In regard to student acceptance, the administration re- 
quested that all selected students voluntarily elect to participate. 

Since the majority of freshmen- at FSU preregister in the summer for 
the fall term, we selected a pool of 100 students enrolled in Hiysics 107 
and not participating in other freshman research projects such as 
the Group Cluster Living Project. These students were contacted by 
mail; and 677. responded favorably, 6% unfavorably, and 27% failed to 
respond. Thirty of the favorable respondents were randomly selected 
and notified. Due to course changes and a misunderstanding concerning 
the one-credit physics laboratory course, seven of the students dropped 
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from the sample. Since classes had met for two sessions, we^ decided 
not to replace these seven dropouts, although there were numerous 
volunteers available. Ke have no reason to believe that our sample 
of 23 students is biased or nonrepresentative of the students enrolled 
in ihysics 107. The comparison groups were selected by matching sex 
and aptitude entrance scores on the Florida Twelfth Grade Examination 
for each participant and randomly sampling from the pool of available 
conventional course students." 

Course Progress , After a brief introductpry and explanatory 
session on the first class day of the fall term, each self-scheduled 
his progress through the course. The mid-term exam was given after 
Lesson 13 and the CAI review bn, the 1440 CAI- typewriter system. The 
reviex^ session was -identical to that offerea to the partial CAI group. 
Since there were tx^o review sessions and: txi^p examinations, the course 
consisted of 33 sessions. Figure 3 presents the cuc^x^lative lesson 
progress curves for the fastest :and slowest student. . 

Insert Figure 3 about here 

These curves are typical in that each student tended to cluster his 
instruction within certain x^eeks. Table 1 .illustrates this cluste..ing 
phenomenon in that 62% of the lessons x<rere taken, .in multiple-lesson 
sessions. There x^ere marked drops in atitendaace during; the mid-term 
exam period and during certain big, extracurricular .eyepts like key 
football games. The students acknox^ledged, during the.ir interviex7S, 
that they utilized their control of the physics instructional schedule 
to optimize their participation in other collegiate activities and 




105 



academic requirements. The studen- ‘ considered this Scheduling 
flexibility to be one of the key benefits of the CAI course. 



Insert Table 1 about here 



The mean data of completion was 10.9 weeks in the eleven- 
week term. Vhen students are allowed to self-schedule, the course 
completion outcos« tends to contradict the co^o, claim that CAI courses 
will dramatically reduce the length of academic terms. On the other 
hand, the mean time to complete the 29 lessons was 23.8 hours of 
instruction. This represents a 17% savings in instruct^<®al time, 
considering the fixed durations of the films, and audio presentations 
plus the opportunity for repetition of difficult material, this 
time savings indicates a significant savings in instructional time. 

It is worth noting that only 3% of the informational presentations 
were voluntarily repeated while approxis^tely 45% of the CAI interactive 
responses were repetitive attempts to seek a correct answer. Thus, 
the prediction of an instructional time.savings for CAI was sub- 
stantiated in this first field, evaluation. 

n>...rse Procedures . In reviewing, our operational 

procedures, we- noted that 95% of the students..' questions related 
■ to the location and oper.ation of the non-computer, audio-visual 
equipment. Without a doubt, the 15om. sound films proved to be the 
biggest scheduling . and operational problem area. While equipment 
failures were minimised by having extra equipment, the effectiveness 
■of operation could only be maintained by having knowledgeable 
: technicians available to set up the films. , Moreover, the number of 
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film failures indicates to us that perhaps video tapes may be a more 
effective approach to solving equipment reliability problems. 

In regard to 8mm concept films, ue discovered that our 
introductory and explanatory materials were ambiguous at times. 

Student coranents and learning performance were both lower and more 
variable for films in general. We are presently investigating tech- 
niques to focus attendance in concept film viewing and increasing the 
amount of interactive questioning interspersed x^ith the viewing. 

X^hile the operations of the course required constant 
attention and service, they were successful in that no student sessions 
had. to be cancelled. Improvements in computer terminal equipment 
xmdoubtedly will further resolve these logistic problems. 

Performance Results. Turning now to the learning out- 
comes, the final grade assignments for the three groups indicate a 
marhed superiority for the autonomous CAI students as illustrated 
in Table 2. Using the sum of the mid- terra and final exam scores, a 
correlated "t" test indicated that the autonomous CAI group was 
statistically superior xdiile the difference between the partial CAI 
and conventional students x^as not significant. The high proportion 
of "A" grades in the autonomous CAI group represents one of the few 
instances where the upper half of a score distribution shifted under 

- i ' 

CAI treatment. It is far more frequent' bhe lox^er half of the 

grade distribution to be truncated due to a CAI treatment. As other 
analyses v^ll support, x^ attribute this superior examination perfor- 
mance to the impact of the CAI conceptual inteYabtifli sections found 
xn.thin each of the lessons. 
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Insert Table 2 about here 



The comparison of performance on different categories 
of lesson materials, mth the prior baseline data collected during 
the preparation of the course, provides some insight into the impact 
of the autonomous CAI lesson material. As indicated in Table 3, we 
categorized the CAI lesson material into three types of instruction: 
(1) assessment of textbook reading, (2) assessment of film presenta- 
tions, and (3) conceptual assessment via problem presentations. For 



simplification purposes, we grouped the lessons into the five main 
concept domaihs'of scientific measurement, optics and light, force 
and energy; electricity, ahd“modem atomic physics. In the last column, 
we included the baseline ' physics pfoblem results collected during 
the developmental phases of tie project. As the baseline results 
indicate, the conventional cburse performance is marked by a gradual 
decrease in achievement while the conceptual complexity is judged to 
be increasing; The performance on the film presentations also 
indicates a gradual decrement in mastery. On the other hand, the 
periormance on the textbook assessment and the conceptual problem 
■ ’exercises remains markedly 'constant. We interpret these stable 
performance levels as indicating k clearer reflection of the performance 
that resulted in the superior examination scores by the autonomous 



CAI group, 



Insert Table 3 about here 



ion 



Using multiple regression techniques, we regressed the 
category lesson scores for the first half of the course onto the 
mid-term examination score and scores from the second half of the 
course onto the final examination score. These results are presented 
in Table 4. The CAI conceptual problem spores, yield significantly 
higher multiple correlations with she examination scores. We interpret 
these higher associated relationships pins the stable performance 
levels for the CAl conceptual problems as indicative of the positive 
impact of the computer interactions , on the examination performance. 
These performance results demonstrate for us that a. CAI approach can 
eventuate in superior conceptual mastery in physics. 






Insert Table 4 about here 

Before this conclusion appears to be too svreeprng, we 
hasten to report that performance on some of the CAI lessons, as for 
example the topic of electrical; inducation, was far from satisfactory 
(the mean correct response -proportion was close to .40). The per- 
formance on the film materials indicate?. -to us a need for extensile 
revisions in the course. .But, ,ii -viewhl-^as a first field trial, the 
learning performance results do support the conception that a computer- 
based multi-media course that attempts to individualize instruction 

can eventuate in superior concept mastery. 

Results. .The attitudinal responses of the 

students to the Brown Scale on Attitudes Toward Computer-Assisted 
instruction indicated a moderately positive reaction to the course. 
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Without reviewing specific .items, the students indicated an awareness 
of the- coristrainted dialogue of the CAI materials, tended to guess 
at times, plus a desire for even more individualization. All of 
the -participants considered the CAI course to be preferable to their 
cofres ponding conventional courses. 

The personal interviews did reveal two especially important 

factions. First, all of the participants indicated a greater sense 
of concept mastery in comparison to their peers. For example, the 
participants claimed to be better explainers of home^k problems in 
comparison with dormimates who attended the conventional course. 

The second reaction relates to the man-:mpchine interface issue. All 
of the participants indicated a pre.ference for the automated type- 
OTiter interaciion in comparison with the CRT-light pen interaction. 

• mie many factors may underlie this unanimous reaction (the flexibility 

• 'and meaniiigfulness of- thft typewriter-presented review material, che 

opportunity to obtain-a personal prpblem-respons ?opy, etc.), this 
■finding Should be investigated iji. light of higher costs associated 

with CRT terminal equipment* . . 

. M=.npnwer— Cost Factors. Cost effectiveness analysis in 

"" educational technology is fraught witji categorization and estimation 

pfbblems. Foremost for CAI projects, one must clearly differentiate, 

if possible, between course developmental costs and the operational 

costs of instruction. . Table 5 presents a simplistic breakdown of 

■ our cost for the college physics project. Most category items are 

self-explanatory. The most noticeable discrepancy is the modest 

cost for actual course operation and the high costs for computer 



systems programs. To date, most CAT project reports have omitted 
the reporting of the high costs Involved in implementing a manufac- 
turer's computer system to the status. of acceptable operations. 

While most of the incurred costs for the FSU project may appear 
substantial in nature, the CAI systems cos.t. represents a one-time 
investment required to organize and manipulate-, the learning data so 
■ that project goals, especially those of course revision, can be 
accomplished. 

Insert Table 5 about here 

Once the developmental costs are amortized, the operation of a CAI 
autonomous course starts to compare favorably with that of conven- 
tional courses. 

The question of cost effectiveness requires our assessment 
of the worth of the improved learning outcomes reported above. ^Ihile 
we are attempting to scale these utilizations, I leave you with the 
problem of assigning value quantities to the unanticipated high 
proportion of superior grades. Quality education has amorphous 
characteristics that pose serious obstacles to definitxve cost 
analyses . 

Summary . In considering the empirical outcomes from 
this first field test, we are most encouraged by the substantial, 
multiple-correlation relationship between the CAI conceptual exercise 
material and the examination scores that lead to the grade outcomes. 

We are presently investigating a class of linear models that, hopefully. 
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will provide us and each student with realistic probability state- 
ments about course mastery. If the values beccme too low, we 
undoubtedly will prescribe additional practice on a concept. 
Otherwise, we plan to allow the student to self-define his level of 
course proficiency. He' suspect that the element of student self- 
commitment may represent a more viable pathway to optimizing the 
outcomes for an individualized course of instruction. 



FIGURE 1 



LESSON OUTLINE OF THE INTRODUCTION TO 

•c 

LIGHT AND OPTICS 



Oblective : 

Introduction to light and optical phenomena. 

Concepts Previously Needed and Acquired : 

Vectors and vector algebra— a vector is a quantity having both 
magnitude and direction. Familiarity with vector addition. 

Concepts To Be Acquired : 

Light travels in straight lines. 

Four ways in which light may be bent: 

1. reflection— light reflected from a plane surface ^iU have equal 

angles of incidence and reflection; 

2. refraction — flight traveling through two transmitting media will 

experience a change in the path according to 
Snell’s Law sin i/sin r = * 

S. scattering — reflecting or refracting light so as to diffuse it in 

many directions; 

4. diffraction— modification that light undergoes when passing the 

edge of an opaque body. 

Properties of light and optical phenomena: 

1. images — ^visual counterpart of an object formed by a mirror or 

lens; 

2. real images— light rays appear to converge at the image; image 

may be detected on an opaque surface; 

3. virtual image— no light rays actually pass through or originate 

at the image; 

4. inverted and perverted images — 

perverted — right and left sides of image 
interchanged; 

inverted — top and bottom of image are 
interchanged. 



Ability To Ans wer the Following Quest i ons : 

VTiat is the relationship bet^veen the angle of incidence and the angle 
of reflection for light reflected fr ''m a plane surface ? (They are equal. ) 

What are the characteristics of the two basic types of images ? (real 

and virtual) 

What are the describing characteristics of inverted and perverted 

images? 

Describe four ways in which light can be ’’bent”. 
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FIGURE 2 
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Remedial 



Lessons 



FIGURE 0 

CUMUIATIVE PROGRESS CURVES OF LESSON 
COMPLETION DURING ELEVEN WEEKS 
OF THE COURSE 




— Fastest Student 

— Slowest Student 
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table 1 



DISTRIBUTION OF COMPLETED LESSONS BY 
ATTENDANCE SESSION 

Number of Completed Lessons Per Session 

1 2 3 4 5 6 



Frequency 


251 114 40 10 3 1 


Percentage of 
Lessons 


.38 -34 .18 -06 -02 -01 .01 




TABLE 2 

FREQUENCY DISTRIBUTION OF GRADES FOR THE 



THREE INSTRUCTIONAL GROUPS 







Grades 




Mean 

Grade 


Conditions 


, A 


B 


C 


D 


CAI Total 


11 


6 


6 


0 


3.22 


Partial CAI 


6 


7 


10 


0 


2.83 

§ 1 " * 


Conventional 


4 


5 


13 


1 


1 • 

2.52 
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TABLE 3 



MEAN CORRECT PROPORTIONS' ON FIRST RESPONSES TO 
DIFFERENT LESSON MATERIAL CATEGORIES BY 
CONCEPTUAL TOPICS 



Concepts ■ 


Textbook 


Films 


Conception 

Exercises 


Base- 

line* 


Scientific Measure 


.698 


-611 


-586 


.591 


Optics and Light 


.733 


.675 


.673 


.578 


Force- and Energy- 


, 706 


.547 


.666 


.483 


Electricity 


.703 


.476 


.653 


.391 


Modem Physics 


.703 


.486 


.605 


.412 



♦Data collected on prior student groups. 
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TABLE 4 



MULTIPLE COERELATIONS OF LESSON CATEGORIES 
WITH EXAMNATION OUTCOMES 





Mid-Term Examination 


Final Examination 


Textbook 


.605 




.694 


FilmS:,, .. 


.587 




.445 


Conceptual Exercises 


.870 


- .. 


.901 




o 
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TABLE 5 



COST ESTBIATES OF DEVELOPMENTAL AND 
OPERATIONAL ACTIVITIES EXPENSED 
BY THE FSU PHYSICS PROJECT 



Man Years 



Category . . 


of Effort 


Cost 


I. Physics Course Preparation 

Physics Writers 
Physics Faculty Consulting 
Films and Audio Preparation 
CAI Coding . . _ 

Behavioral Scientist 


2.0 

1.0 

.5 

_ 1.0 
i7o 


$ 20,000 
20,000 

4.000 

6.000 
14,000 






$ 64,000 


II. Computer Systems Development 

Data Management System 
Data Analysis Programs 
Data Analysis Operations 


3.0 

2.0 
1.0 


$ 30,000 
16,000 
6,000 






$ 52,000 


HI, Administration and Services 
Administrator 
Secretaries 


.5 

2.0 


$ 7,500 

10,000 






$ 17,500 


IV. CAI Course Operations 
Proctors 
Physics Tutors 

Computer Operator and Technician 




$ 2,500 

1,200 
3,000 






$ 6,700 




TOTAL 


$ 139,200 



Definitions and Examples as Feedback in a 
CAI Stimulus Centered Program 



2ert Keats 

The Florida State University 

The requirement for correctional feedback to erroneous ai^swers in 
to facilitate acquisition has bean well established in verbal lemming and 
pedogogical experiments. Unfortunately, the precise form and content for 
these correctional messages that ^n.11 maximize learning remains a cloude 
For the classroom teacher, restating the problem, giving analogous examples. 

Citing the solution rule in a verbal form, cr just giving the correct answer are 
commonly observed. Witiiin the tradition of programmed instructional research, 
the role of prompts or hints as correctional messages has led to a^iguous 

iT • 

findings . 

Programed Instruction (Pl7 ' advocates of •■Stimulus-Centerpd Programming" 
(Klaus. 1965) place a stronger- SAphasis on correctional procedures for erroneous 
responses than on reinforcement of correct responses. More currently, authoring 
within Computer Assisted Instruction (CAI) manifests many of the obvious alternates 



to correctional approaches. 

The present stud^ considers the case for a specific ayea-proofs in mathe- 
matics. Correctional feedback alternatives in this content area are usually a 
formal definition (symbolic statement of a rule), a verbal definition (an English 
language form of the formal definition), or a numerical example of the formal 
definition. The precise nature of the feedback tends to represent the artfulness 

and style of the course author. 

The present study was designed to test the relative efficacy of the latter 
two feedback modes, verbal definition and numerical example. In addition to 
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assessing the opposed .corvditions of definition and example, we developed a more 
detailed CAI program that utilized both message ty:;a3 according to the best 
intuitions of an experienced mathematics instructor. This combination treat- 
men" represents Dne conception of a conventional classroom approach to mathe- 
matics instruction. 

In COP .paring the use of verbal rules and examples as aids in mathematical 
problem sol/ing, the present study is related to previous research concerned with 
"inductive vs. deductive learning", "discovery vs. expository learning", and 
"rule vs . example learning" (e.g., Belcastro, 1961; Gagne'^ & Brown, 1961; Guthrie, 
1967; Yrrjmboltz & Yabroff, 1965; Michael,. 1949; Scandura, 1964; Sbhel, 1954; 
Worthf-ti," 1967) . However, this study differs from them quite markedly in that 

ff 

the * rules" and examples are .vised solely in CAI-feedback as an' aid in justifying 
the steps in a mathematical proof and not as "the technique" for problem solving. 

Jensen (1966) inyeatigated the effectiveness of CAI instructional unita 
involving algebraic proofs,. ,.Ss ^ 7 .ere asked to verify simple algebraic relation- 
ships by providing the necessary steps and apply the appropriate axiom to justify 
each step. One group of Ss was presented with the complete set of axioms to be 
used in the proofs. They were then shown sample proofs demonstrating uses of the 
axioms and were then given the task of proving a collection of thbbfms using the 
given set of axioms. .. For the other group, the set of axioms was' pattitionad 

I " r 

into several subsets.. For each subset, sample proofs were given to illustrate 
the application Of. the axioms in the subset and Ss were required to prove 
theorems using axioms, .fro^tt the subset before preceding to the next subset. Mean 
differences for the two groups with respect to program performance scores, post- 
test scorfes, 'and program times were not significant. Large differences between 

’ - 

pretest and posttest scores for all Ss indicated that CAI instruction was quite 
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effective in teechins some basic ideas in deductive reasoning. 5Jo correctional 
feedback was used in the Jensen study and Ss were required to construct the 
steps in the proofs. In the-present study. Ss were only required to select the ' 

appropriate rule which justified each stop in the proof. 

Easley, Gelder,. and Golden (i9W) using the PiAIO system at the University 

of Illinois, developed an iaganious..t’«orem proving program »*ere the student 

^ • r * 

could' s'tructure-'l'ixc o.wn oy -t> • : , 

. : rnn"'-’ Grt fo“a. To our knc'-ricdge, no hypotheses have 

step, chaiiging xt to cn eqa.-vc.a.«:t.i: a.o_ui, . ^ . .. . 

been tested using this program. ... . 

In sugary, this study was intended to answer the following questions: 

.IT" 

(cl) Should verbal definitions or numerical examples be uss'd" 
nz che form of CAI correctional feedback- as. applied :ta^ 

steps in mathematical proofs? 

(h) Should a CAI instructional unit involving mathematical 
^ ^ pr^^s be structured in much the same Way as the classroom 

unit is developed? 



< 5 * 



' - . j ... 



] 







Method 



Subjects 

Forty- five student volunteers were selected from the ninth grade of two 
Tallahassee. Florida, junior high schools. There were 18 boys and 27 girls. 

14 to 15 years of age. These students had just completed the year-long Algebra 
I course, and were familiar with the real number system and the concept of a 
mathematical field for real numbers, including the terminology used in describing 
the field properties (commutativity, associativity, etc.).* No student was 
famili^ with the complex number concept. 

Materials and Procedure . 

Each S was first administered'- the Reasoning sub-scale of the SRA Primary 
Mental Abilities Tests-grades 9-12. Next, each S was given a mimeographed 
list' containing 24 brief descriptions of mathematical rules w.th an abbreviation 
for each rule. The Ss were told that they would use these rules in respondxng 
to questions in their CAI programmed unit. The Ss were familiar with most of 
the rules and each of these was briefly reviewed by the author of the progranmed 
unit using a blackboard to illustrate the rule when necessary. It was explained 
to the Ss that the unfamiliar rules in the list would be made clear to them 
early in their CAI programmed unit. These unfamiliar rules involved the set 
of complex numbers and were subsequently introduced in the programmed unit. 

Using the blacicboard, Ss were instructed in the use of the IBM 1050 
Terminal (a typewriter-like device) so that they could "sign on" the IBM 1440 
System and make responses \dien directed to do so. Each S was then taken to a 
terminal and given assistance in "signing on" and re-instructed in the response 
procedures. The CAI programmed unit was presented to Ss at the terminals. In 
some cases, due to the length of the unit, it was necessary for Ss to complete 
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S da 

computer Correct. 

computer (a+(c+e),b+(d+f)) = ((a-te)4e, (b-W)4f ) 

S da 

computer No. Numbers are added two at a time. This rule tella us that it 

makes no difference. .y^ich two are added first. Try again. 

S ca 

computer ^No. Numbers are added two at a time. This rule tells us that it 
makes no difference \diich two axe added first. Try again. 

S aa 



L 



computer Correct . 

Group 11 - numerical example. The programmed unit was exactly the same 
as for Group 1 except that correctional feedback was in the' form of a numerical 
exmiple. 

Using the same two steps as in the example' for Group 1, a typical student^ 
machine interaction for a S in Group 11 would be as follows: 



computer 



(a,b) + (c4e,d4f) = (a+(c4e),b+(d4f ))’ 



am 



computer 

S 

computer 

S 

computer 

S 

computer 



No. Why does (6,8) + (3,4) = (9,12^) ? Try again. 



aa 



No. Why does (6,8) + (3,4) - (9,12^ ? Try again. 



tr 



No. Why does (6,8) + (3,4) = (9,12) ? Try again. 



da 



Correct. 



computer (a+(c+e),b+(d+f)) = ((a-fc)-fe, (b-fd)+f ) 



dc 



ERIC 

hiwmiifftffiniiaaij 
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computer 12+(13+14) = (12+13 )+14 is an example of what happened. Try again. 



ca 



computer 12+(13+14) — (12+13 )+14 is an example of what happened. Try again. 



S aa 

computer Correct . 

^ Group r.ir . 7 detailed program with combination feedback. These Ss received 
a lengthy introduction to complex nunbers which included formal definitions of a 
complex number and ecjuality for complex numbars. Both of these definitions were 
accompanied by questions and short numerical problems. There was a detailed 
review of the field properties for real nirribers and a rationale for the necessity 
of complex numbers as well as a preview of what was to be done in the program- 
The sW Were asked "to respond quite often in this introductory section (29 times). 



Following incorrect responses in the introduction, the problem was explained, 
and the answer was given to S. Before each proof was presented, a few short 
problems and questions attempted to lead S to discover the objective of the 
following proof. When’ the definitions of addition and multiplication of complex 
numbers were introduced, practice exercises were used to allow S to become quite 
familiar with these operations. There were four exercises in addition and eleven 
exercises in multiplication. Again, following incorrect responses, the problem 
was explained and the answer was provided for S. The task of supplying rules 
to justify steps in the 11 proofs was the same for Ss in this group. However, 
feedback varied in form between the verbal definition and the numerical example. 



The form chosen for a particul .r step was the one expected to be more effective. 



Verbal definitions accounted for the feedback on 36 of the steps and numerical 
examples were used as correctional feedback on 28 steps. For a given step in a 
proof, the feedback was identical to feedback on the same step for either Group I 



or Group 11. 



the unit at a later date. This was done within two to five days of the first 
visit. After completing the CAT progrsniaed unit, each S was given a test con- 
sisting of 10 True-False, and 10 Multiple Choice questions about the information . 
presented in the CAI progranned unit. Most of the True-False questions 
transfer items with a few recall items. Eight of the ten Multiple Choice items 
were taken directly from a step in one of the proofs from the CAI programmed 
unit. The other two were a transfer and a recall item. This test had an 
internal-consistency reliability of .38. 

CAI Programmed Unit 

The unit consisted of a series of 11 mathematical proofs. These proofs 
establish that the set. of ccmiplex numbers under the operations of addition and 
multiplication is a mathematical field. As each step in a proof was presented, 

S had to respond by keying in the abbreviation from his list which represented 
the rule justifying that step. Since there were 24 rules from which to select, 
the completely naive S had a probability of slightly greater than .04 of keying 
in the correct rule on his first trial. Following an incorrect response, S was 
given correctional feedback and asked to try again; if he failed again, he was 
given the same feedback and asked to try again. This process was repeated until 
the correct response was entered. With the exception of a few steps ^ere more 
than one rule was an acceptable response, the naive S \jho kept account of his . 
erroneous responses increased his probability of success by approximately .04 
after each incorrect response. It was hoped, of course, that the correctional 
feedback would reduce the domain of possible correct responses so that no S 
would be playing a guessing game. The 11 complex number proofs contained 64 
steps; three of the proofs had only three steps and one proof had nine steps. 
Since a complex number was defined as an ordered pair of real numbers, the rules 
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used to justify t'he steps in the proofs were largely concerned with the field 
properties of real numbers and with algebraic Operations on real h(tmbers. 

Design - '..c. i • i ■' « ' 

The Ss were randomly assigned to one of three CAI instructional treatment 
groups (15 Ss per group). The three groups differed in method of introductory 
instruction nnd in the type of feedback following aii incorfecf response. 

Group-.T T- 'Verbal definition. The introductory material included formal 
definitions for: (a) .2 complex number, (b) equality of complex numbers, (c) 
addition of complex numbers, and (d) multiplication of complex numbers. A single 
example illustrated each definition. There was also a brief review of the field 
properties .ftfr real numbeiss. . After' the introduction, which took approximately 
nine minutes. jof computer time," the' 11 proofs werh presented consecutively.' "No 
review or preliminary' -information was given between proofs . There were no pro- 
vision^ -for the. Ss to respond in the inttoductory section. Ss responded only to 
steps iai-the proofs.- - 

An 'Sample of two consecutive steps from one of the proofs with several 
incorrect responses on the part Of S follows: 
computer (a,b) + (c-fe,d"ff) = (a-f(c+e),b+(d+f )) 

S • --ca 



computer' ‘f- No. We" combined tw6 complex numbers into one complex number. What 
rule allows thi<?? 

i- . ■-> . * .• ; 

S dc 



computer No. We combined two complex nu.abers into one complex number, 
rule allows this? - . ‘ 



What 



dm 



computer No. We combined two complex numbers into one complex number, 
rule allows this? 



What 



Results 



Performance Differences -Programmed Unit 

A repeated measures analysis of variance using errors per step per student 
as the dependent variable indicated a highly significant group-by-proof inter- 
action (Table 1). Figure 1 shows that most of this interaction can be attrib- 
uted to the performai.ce of the verbal definition (Group I), and the combination, 
(Group III). Group differences are not easily discernable in cases involving 
such large interactions. Therefore, Group III was removed, and another 
repeated measures analysis of variance (Table 2) indicated a significant group 
difference between Groups I and II as well as significant interaction. 



TABLE 1 



ANALYSIS OF VARIANCE-I 



Source 


df 


MS 


F 


Group (I) 


2 


64.33 


2.91 


Pr‘;>of (J) 


10 


27.30 


9.57 * 


Subjv*ct within Group (K(I)) 


42 


22.06 




IJ 


20 


7.71 


2.80 * 


JK(I) 


420 


2.84 





* p <.05 



o 

ERIC 



129 



.GR310RS BY FilOOP 



oei 








I 

i 

i 

• 

I 

i 

i 

{ 

I 

I 

t 

i 

I 



CV 



00 



a 



o 








-H 
























a 








•r4 














o 








cd 


Q 






•Q 




a 


-H 


'O 


r-4 


f-i 


*> 


CO 




o- 




CD 




g 


E 






cd 


cd 




CD 


« 


M 


a 


> 






o 






o 


•i-e 






}-l 


4J 




«o 


d4 


td 


tj 
»— t 




'V 


c 

•<rl 


a 


o 


G) 


•o 


CIS 


cd 


T— 1 


s 


•O 


.Q 


•rl 


o 






cd 


o 


G) 


a 


AJ 




C) 


0) 


<D 








Q 








H 






M 


H 




M 


M 


M 








P4 




3 


zt 


D 




O 


o 


O 




J-i 


u 


U 




e> 


o 


o 





■ O x<l 







:jii5pn:}S ^s:JS -isj sxoxx^- 



o 

ERIC 

hiaifiiifftaiTi-Taaa 




TABLE 2 



ANALYSIS OF VARIANCE-II 



Source 


df 


MS 


F 


Group (I) 


1 


123.13 


4.60 * 


Proof (J) 


10 


19.10 


6.3? * 


Subject within Group (K(I)) 


28 


26.79 




IJ 


10 


11.02 


3.67 * 


JK(I) 


280 


3.01 













* p (. 05 



V - 

A sign test for to*-al nuinber of successes oh the first trial for each of 
the 64 steps in the proofs indicated that Group I was significantly better than 
Group II (p (.001), Many of the first pass successes for Group I came in the 
last two-thirds of the program. In fact. Group I was greater than or equal co 
Group II in number of first trial successes on 32 of the last 38 steps. On 
each of the last six proofs. Group I correctly identified approximately 1051 
more of the rules on the first trial than did Group II. (Figure 2). 



Insert Fig. 2 

Average times to complete the program were nearly identical for Gfoups I 
and II (Table 3). Group III averaged one hour longer at the terminals because 
of the more voluminous nature of their program. 



FIRST PASS SUCCESS BY PROOF 
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Pre and Posttests. 

Mean scores for the three groups on the pretest (Primary ilental Abilities 
* 

Reasoning Scale) were quite similar (lable 3). No significant differences were 

found on mean pos'ttest scores for the three groups. There were significant (p(^.01) 

negative correlations between cotal errors on the programmed unit and posttest 

scores for both Group III (-.655) and for all the groups combined (-.429). 

« 

. Discussion 

3 

" I 

With respedt ^o GAI presentations of mathematical proofs, it appears that 
providing correctional f eedfesck -in the Iform of a verbal definition is of more 
benefit to the learner than using a numerical example. The resu_t is compatable 

i 

with a finding by Wittrock (l963) which related that giving a rule was more 
effective than not givi g a rule. This result is also in accord with the belief 
of Ausubel (1963 ) vdio states : 

Providing guidance to the learner .in the form of verbal explanation of 
the underlying principles almost invariably facilitates learning and retention 
and sometimes transfer as well. Self-discovery methods and the furnishing of 
completely explicit rules, on the other hand, are relatively less effective. 

(p.56) *■ 

Figure 1 indicates that most of the interaction between Groups I and II 
occurred early in the program- at proofs three and four. This interaction can 
be explained by an unfamiliarity with the rules- Ss were in a confused state and 
were applying rules for the first time.' As Ss progressed through the last half 
of the proofs, and rules were reapplied, differences were more pronounced. Given 
that the learner is in an erroneous state, providing correctional messages in 
the form of a verbal definition seems to facilitate certain associational 
mechanisms so that probabilities for success on later trials involving reap- 

i 

plication of the same rule or application of a similar j rule are greater. A 
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glance at Figure 2 will support this point as a rather constant probability of 

success is observed over the last six proofs. 

The performance of the group receiving a treatment designed to approximate 
the classroom approach (Group 1X1) was somewhat below expectations. The longer 
and more detailed treatment of mathematical proofs within the CAl framework may 
be unwarranted. 

Scores on the posttest were quite low for all three groups (Table 3). The 
C /VT progrsmmed unit was much more difficult than ma».iy other CAl programs avail- 
able for the same age group. The large number of errors made by all Ss was an 
indication that the unit was also more difficult than other mathematical learning 
materials to which the Ss had been previously exposed. The small number or items 
and the extreme difficulty of the test contributed to its low reliability. 

The very poor performance of Group 11 on proofs six and nine (Figure 1) 
might be explained by the nature of the steps involved. Proof six was the first 
proof concerned with multiplication concepts. These concepts were not used in 
the fir&t five proofs. Apparently, the Group 11 Ss had much more difficulty in 
adjusting to the transition from the addition proofs to the multiplication proofs. 
Proof nine was the first proof involving steps where both sides of an equation 
were operated upon (e.g. multiplication and subtraction of terms on both sides 
of the equation). It is suspected that the numerical examples were not very 
effective in illustrating operations performed to both sides of an equation. 

The results of a recent study by Scandura (1967) suggest that CAl branching 
decisions consider more specific forms of response analysis and contingent feedback 
rather than more general measures such as number of errors, average latency, etc. 
Scandura advocates a need for determining appropriate feedback in a more efficient 
manner than what has been done previously. The present study represents an attempt 



135 



to identify appropriate feedback in a specified area-iuathematical proofs. 

‘The effects of feedback by verbal definition sees to vary with the com- 
plexity of the learning materials. The results of this study indicate that wac.. 
the learner is in 'an erroneous state, providing correctional messages in the form 
of verbal definitions increases his probability of being removed from that state 



more than the use of num ‘:i'caV examples. Taf themore, verbal definitional feed- 
bade enhances the prcbii'ility of inia.ediate success on later trials involving 



re- application of the same rule or application of a similar rule^ 
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The Developioent of an Academic Year Institute in 
Computer-Assisted Instruction 



Henry T. Lippert 
Institute of Human Learning 
Department of Educational Research 
Florida State University 



Background _ . 

In progresjr-at the Florida State University in Tallahassee, Florida, is a 
y 0 ar-long institute for the education of graduate students in Computer— Assisted 
Instruction (CAI), This program is’ funded through Title VI, Part B, Faculty 
Development Programs, of the Higher Education Act of 1965. The overall objective 
of the institute is to provide an opportunity for pre sen t ard prosp^tive faculty 
to develop an in-depth understanding of all aspects of CAI. The institute began 
September 15, 1967, and wi*l:i-ehd^uneJt5.>_1968i. 

As anyone who lias tried to recruit faculty in the area of CAI can testify, 
there has been and is. still a critical shortage of adequately trained faculty in 
this area. It was (and still is)-±he contention of the directors of this program 
that the short-course or workshop approach to learning about CAI is inadequate and 
insufficient. The promise offered by the use of modern digital computing "equip- 
ment in education is jeopardized by attempting a short-course approach to training 
personnel in the area of CAI. 

Program Objectives 

We are therefore attempting to develop this in-depth understanding through 

-m 

a series of theoretical courses offered for grada?te' credit, individual projects, 
hands-on experience with computer systems, and close personal interaction between 
the students and the CAI faculty. The program has been guided by a series of six 
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overall objectives. They are: 

1* To acquaint the particijants. with the field of ComputerrAssisted 
Instruction^ This includes a survey of existing electronic equip- 
ment, computer languages that operate various CA.I systems, and 
various applications of CAI at the collegiate level. 

2. To provide the participants with an understanding of the educa- 
tional and learning theories that form the basis for the use of 
CAI. This includes coverage of learning theories, test theory, 
optimization models, ^yid dynamic programming; and in addition, 
the interrelationships ^f learners’ aptitude and characteristics 
and the structuring of collegiate course’ content in order to best 
reveal how CAI can individualize instruction. 

3= Tq develop a high degree of proficiency on the part of the partic- 
ipants in the utilization of the FSU Computer -As sis ted Instruction 
systems. This includes, an operational understanding of the hard- 
ware systems, how they are run, how to program each, and how to 
obtain sophisticated data analyses. 

4. To develop knowledge about the newer techniques of data analysis 

such as sequential analysis, test of goodness-of-fit for instruc- 

* * * * 

tional models, and dynamic decision making. 

5. To learn how to administer a CAI installation so that the partici- 
pant will understand the operational factors of cost, time scheduling; 
course development, and personnel. 

6. To author a CAI course segment on one- of the FSU systems and to 
evaluate its efficiency. 



o 
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GAI Center Facilities 

At the Florida State University, the graduate college contains several 

• • 

institutes which are primarily research oriented agencies within FSU %dierein 
faculty members are provided with space and appointments \diich allow for a 
maximum of research aPiCi a minimum of teaching responsibilities* 

History 

— - «' i . 't 

The Computer-Assisted Instruction Center is a part of the Institute of 
Human Learning at Florida State. 

Computer-Assisted Instruction at the Florida State University started with 
the installation of a single IBM 1050 typewriter terminal in September, 1964. 

This terminal was connected by a long-distance dial telephone connection to an 
IBM 7010 computer at the Thomas J. Watson Research Center in Yorktown Heights, 

New York. 

After a yea:: of operation on this remote basis, an IBM 1440 computer with 

, .... " ^ ^ 

five terminals was installed in the Computer-Assisted Instruction Center. Several 

j ■ • • 

instructional units were developed including the topics of Test Statistics, 
Chemical Concepts, Modular Arithmetic, Scientific Notation, Prevocational Literacy 
Training, Concepts of Social Welfare, and Physics. 

September, 1966, the operation of the Center was expanded and additional 
full-time faculty, Drs. Duncan N. Hansen and Walter Dick, and additional staff was 
added. A major effort was marshalled toward the development of a college level 
course in Physics. Aidditional course materials were developed in the areas of 
Sequential Testing, Applied Statistics, Computer Language>^ and another major 
project in Junior High Science was initiated . 

The faculty and staff were again expanded for the 1967-68 school year when 
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Drs. Ifenry T. Lippert and Richard Lee joined the institute, and an additional 
computer system was installed. This new system, an IBM 1500 Instructional System, 
provided eight terminals and the expanded capabilities for display, timing con* 
trols and response recording over the 1440 typewriter system. By the winter 
quarter of the 1967*68 school year, this system was expanded to 30 instructional 
terminals. During this school yeai. Physics 107 was taught for credit on the 
1500 system. Both systems were used by students to review and study physics in 
preparation for their examinations. 

Space and Personnel 

The Codiputer-Assisted Instruction Center is currently located in converted 
space in the basement of tne men'^s 'gymnasium. The Center contains approximately 
4,100 gross square feet. There are 4 full-time faculty members, 5 full-time 
professional staff, 12 semiprofessional staff, 3 secretaries, 6 research assis- 
_ants, 3 student assistants, and 19 graduate trainee fellows— a total of 52 
the Center •. In addition, there are two complete computer systems 
which include a total of 36 instructional terminals. Three areas of space are 
defined for general use, two audio-visual rooms and a duplicating and storage 
area, Motxon picture projectors, tape recorders, dial-access telephones, and ' 
laboratory work space are also in use. Supporting equipment for the computers 
includes key punches and-^ card sorter. 

Selection and Prior Background of Trainees 

Subsequent to the announcement of the scheduled institute, approximately 
190 inquiries were received. In response, some 109 applications and further 
information were sent to the inquirers who appeared to be most promising. The 
Center received 87 applications and of these, 53 were judged to be well qualified 
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for the prcgran. The Center offered admission to 20 of these students and all 20 
accepted and enrolled. A total of 39 dependents were included, a ratio of 2 
dependents per. trainee. 

All of the participants had bachelor degrees and 14 held master's degrees 
at the beginning of the institute. The remaining six will complete the academic 
work necessary for a master’s degr c by the end of the summer quarter, 1968. five 
of the participants will be cojipletiEg the Ph.D, degree by the end of the suraner 
quarter, 1963. 

Course Work 

During the three quarters of the institute, the oarticipants will accumulate 
45 quarter hours of graduate course. woxk. Eight courses are required for the 
trainees and are as follows: 

ENR 537, TEC HNIQUES OF PROGRAMMED INSTRUCTION : An historical review of the 
development pf prograsimed instruction and important research relevant to instruc- 
tional variables is covered in this .course.. Programming techniques, styles, and 
methods of course development are covered mthin the course. Special emphasis is 
given to author techniques. 

ENR 5^:4, ORIGINS 0? IN DI VIDUAL DIFFERENCES IMP O RTANT IN EDUCATION ; A survey 
of biological, psychological, and social origins of individual differences as 
they apply to instructional problems is given. The applicatipn of individual 
differencas especially pertinent to aptitude differences are viewed in terms, o.f. 

educational and instructional strategies. 

" • . 

ENR 565, HiniAN FACTORS IN TR AINING AND INSTRUCTIONAL SYSTEMS: A survey of 

i ' • « * . ^ II “ ‘ 

the systems analysis approach to applied training and instruction systems with 
emphasis on the 5.nter-relationship among human factors, technological resources. 
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and perfornance goals. 



ENR 530, ^IgbTER:ASSISTED BTSTRUCTIOK : A survey of all existing CAI systems 
is provided and an intensive introduction to the ?SU 1440 and 1500 CAl systems is 
provided as part of this course. The students will learn the operational concepts 
of the systems and will write proprainned materials for the instructional terminals. 

EKR 532, SIS_^7_E0U0AT I0NAL DATA ; A survey of important devel- 

opments in research ralevan^ to education which depends on computer data processing. 
Instruction in the use of multivariate procedures, analysis of variance, and 
regression analysis is provided and extensive exposure to the F3U Control Data 
6-rOO computer systen .is included in this, course x-roirki- ' - - - • 

E!<U 567, IN FORKATIOH P R OCESSES OF T.EARHING AND 

Infprr-.atior: r- ■' :psing models of associative structures and complex" 
symbolic behavior are crncidered as prototypes of future educational models. 

Gomeuter and Monte Carl sinulation techniques are developed. ' ’ 

556, A survey of recent theory aiid research 

reiacing to perceptual ar-.d cor.eitive irocesses is covered. These behavioral pro- 
cesses are in turn related to the Ivisi: structuring of educational material at all 
levels of instruction^ 

ENR 566, a survey of stochastic' pro- 

>.es.ses and dynamic prorra-ming are preseiited as instructional models. Optimization 
techniques arc devalopid- Applicaticnn of muiti-stege real-time decisi on = processes 
within Computer -As. sis ted Instruction ore investigated. 

In adaition to *-hir course work, frequent group seminars have focused on 
apnlicational problccr that are cost appropriate to developing competencies of an 
author and manager of a CAI .system. These seminars are run on- a case-study basis 
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and reflect the individual interests of the participants. 

Individual Project 

During the fir’st quarter, each trainee defined an area of interest to him- 
self, one in which he wished to develop sone materials for presentation by CAI. 

From these individual projects emerged tv/o major integrated efforts. Each trainee 
will. contribute a portion of one of the total projects. 

-first of these projects is the development of a programmed instruction 
course to teach the author language COURSEWRITER II, used on the 1500 system. A 
complete Gagne^ task, analysis of the structure of this language was made by one of 
the trainees. This enabled each trainee interested in this project to choose a 
portion of the total task, and to see how his part relates to the whole project. 

The trainee who is working on this project is also teaching the portion of the 
course in CAI. that deals with his chosen topic. 

The second project deals with the systems approach to education. This 
course is a multi-level introduction that will allow at the first level a one 
hour, overall^ broad introduction to the systems approach. At the second level 
the student will be given a detailed presentation of each of the basic concepts 
requiring about one hour of learning for each concept. The concepts being con- 
sidered are (a) specification of content, and task analysis; (b) behavioral 
objectives; (c) sequencing of objectives; (d) assignment of madia to objectives; 

(e) criterion measures and (f) revision procedures. The third level will deal 
with specific curriculum or administrative areas such as (a) student registration, 
(b) adult education, (c) in-service education, (d) math education, and similar 
topics. The student in this course will be able to study systems concepts in depth, 
across curriculum and administrative areas or will be able to study within these 
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areas across the systeins concepts « 

Two examples of individual projects warrant some concent and illustrate the 
kinds of projects on which the students are working. 

The first is investigatir.g a linear instructional approach to teaching the 
subject of curriculum organization. The three organizations are the traditional 
ones! subject matter, core and activity. The topics have been sequenced on a 
logical ordering of the concepts for one group. ;The second group will be taught 
in a traditional classroom manner. The other group will allow the student to 

■ * :• - ■ j.- 

select topics of his choice according, to his desire. All the groups will be 

■ « I « . ' " . 

given information as .to how. well they performed on a pre“test. This program is 
being run this week and next on the 1500 system. 

The second individual project will automate the routine aspects of academic 

counseling. The first task of this project was to identify the types of informa- 

*■:? * . 

tion exchanged between the student and his advisor; the second is to determine 
what parts could be programmed and the third is to evaluate the., effectiveness of 
the program by actually using it to counsel stud.ent:S. The results will be judged 
as to its adequacy by a panel of faculty members. This prpS^^ will be run on the 
1440 system. 

Future Plans 

The trainees are each working on degree programs and will, in nearly all 
cases, continue through a doctoral program. These programs are in Educational 
Research, Higher Education, Educational Administration, and in Psychology. 

Application has been made for an Institute to be held during the 1963-69 
school year. Word has been received that this Institute will be funded. 



THE EFFECTS OF VERBA.L REINFORCEMENT ON 
COMPUTER-ASSISTED LEARNING^ 

Kenneth !Majer 
The Florida State University 

A * i ;Jhe notion of verbal reinforcement (RF) has been in evidence in 
psychological literihi^ for years , and now with the advent pf Programmed 
instruction (PI) arid Computer-Assisted Instruction (CAI), the in^)lications oi ^ 
this theoretical concept have fa. -reaching pracUcal significance. The generally 

positive results of the effects of RF in verbal learning have led programming 

* - . - 

authors to the foregone conclusion that RF in programimng would have a 
favorable influence <xi learning. The results to date, however, are not as 
conclusive as one would expect. The majority of current investigations either 
conclude or assume that feedback after responses does not seem to be a 
significant variable in lee.ming (Moore suid Smith, 1961; McDonald and Allen, 
1962; Hough and Revsin, 1962), although there is some evidence to show that 
performance in PI is enhanced with feedback and RF (Krumboltz and Weisn^, 
1962; Krumboltz and Kiesler, 1964; Grace and Cantor, 1964). 

. t**- «» 

Holland (1964) suggested that this inconsistency may be due to the 

fact that investigators have not looked carefolly at questions concerning the 

* 

* - 

^Paper presented at the Annual Meeting of the Aimerican JEdi national 
Research Association, Chicago, February, 1968. 
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nature of tbe RF. Gilbert (1962), for example, has suggested that a student 
should receive feedback that is particularly reinforcing to him. My investi- 
gation sought to examine the effects of five separate feedback cooUtions in a 
CAI learning sequence utilizing the programming capabilities of the IBM 1440 
instructional system to store idiosyncratic verbal feedback (i.e. , specific 
words) for each of the 75 subjects involved. The unique capability of the 
CAI system to vary specific segments of course presentation without altering 
the instructional sequence of the program allowed the five feedback conditions 
to be tested with identical leamii^ materials. The program, therefore, 
allowed the students to receive varying degrees of pleasing, personalized 
feedback following correct responses C "To keep from confounding the results 
with interactive effects of particular types of feedback following incorrect 
responses, all five groups received' identical information after each inqorrect 
response given to a question. This information was in the form of a hint , the 
correct answer, or a rule— depending on the* location of the questicm in the 
program — and w»as usually preceded by the word ’Incorrect. ’• 

Method 

Subjects . For this study, 75 students, 37 boys and 38 girls, were 
selected from among volunteers in grades nine through twelve from several 
Tallahassee junior and senior hi^ schools. All students bad con^ileted a 
year-long Algebra I course, and were familiar wifli the real number systen. 
and the concept of a mathematical field for real numbers, including the 
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terminology of field properties (commutativity, associativity, etc. ). No 
student had studi^' coiqplejC numbers in school. 

Materials . ' The first task for each student was to determine his 
individual preferences for pleasing words. To do this, each student ranked 
the five most pl'easihg and the five least pleasing words from a list of 19 
positively affective words. In addition, each student was given the option to 
insert three words of his own if none 6L the words on the list were particularly 
pleasing to him. 

Each student then took the Mental Reasoning portion of the SRA 
Primary Mental Abilities Tests,. Grades 9-12 (PMA). This xx>rti(m includes 
the Letter Series (LS), Word Groupii^ fWG), and Number Series (NS) tests. 
Then, all students received a unit .oneopgqplex numbers which was presented 

• •'*4. 

via the CAI systeiil'. The unit consisted ci a field properties review, definitions 

" 4 • “ m 

of complex number terminology and a se.c^on of 11 proofs , in which knowledge 
of field properties arid hew definitions, was req^i;ed to justify the steps in each 
proof. ■ '■ ■ 

After con^leting the programmed unit, each subject was given a test 
of Ifi, true -false and 10 multiple -choice items. Most of the true -false items 

were transfer questions with a few recall items. Ei^t Of file 10 multiple- 

? 

choice items were taken directly from instructional frames. The ottier two 
.^t items were a transfer and a recall item. The reliability of thi^. posttest 

was .38. (It should be noted here that the uniformly poor'i^rfommance on the 

»• • 

posttest which contributed to its low reliability was due in part not 99 ^ to the 
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small number of items but to the feet that the complex numbers materials are 
considerably more difficult than most erf the CAI learning materials we have 

“■•I 

available. It was felt that differences in RF effects would show up more 

“ * 

clearly if the criterion performance for the subjects was not uniformly hi^, ) 
P rocedure . The students were randomly assigned into five groups 
of 15 each. Each group was to complete the program described above. I>iring 
the proof section of the unit, the student had to determine and enter the correct 
reason for each step in a proof before tte next step was presented to him. The 
student had a Ust of 24 properties of real and complex numbers from which he 
selected the Correct reason for a.particular step in a proof. Therefore, it was 
possible for a student to make 23 errors before selecting the correct reason 
for a ^veh step in a proof , and more if he entered a wrong reason more tiian 
once! Following an incorrect response, the student was given a hint and told 
to ti^ again until the correct response was entered. The program was 
presented at IBM 1050 terminals (a typewriter-like device) throu^ the IBM 

1440 system. 

Design. The five groups differed only in the type of feedback they 

received after a correct response. 

Group I - Common Reinforcement (GRF). Nineteen reinforcing 
words or pairs of words were taken from existing programs written by authors 
at the FSU-GAI Center. These 19 words were randomly programmed to follow 
correct answers. This condition served as a control group in the sense that 
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the occurrence and nature of these words are representative of those found 
in a typical CAI lesson. 

^ ^ * A* * 

Group n - High Eeinforcement (HRF). For each of these students, 
space was made available in the computer core area to store the 
three words which he chose from the list to be most pleasing. In this way, 

ft - ^ 

t h»«» idiosyncratic respomes were available for personalized positive 
feedback.to correct responses in the program. The three responses were 
programmed to be randomly distributed among correct responses in the 

matenals. 

Group in - Low Reinforcement (LRF). For each member of this 

- * "ft 



group, idiosyncratic feedback followed correct responses in the same way as 
d^ribed above for HRF subjects. The only dffiereMe is 1^“®® students 
received words which they had rated least pleasing from the 19-word list. 



G^p W - Knowledge of Results (KOR). Ibis group of students 
received no reinforcing feedback following correct answers. In place of RF. 
they received blank counter displays. At the beginning of flie program, they 
were instructea ttiat they would know when they were correct because they 
would receive either the next question or an instructional statement. If they 

» . w ^ * 



answered incorrectly, as in the other groups, they would know because they 

•- - - . i ; . 1 - 



would be given the correct answer 



or a hint to help them find the correct 



^^wer. . , 

Group V - State Reinforcement (SRF). This group received 

feedback only after every fifth question. At this juncture, they were told 
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how many of the last five questions' they answered correctly on the first try. 
If they answered four or all five correctly, they received the following 
statement: "You answered 4 (or 5) of the last five questions correctly on 
the first try. Keep up the good woiic. " If they answered zero or <mly <Mie, 

f 

two or three of the five questions correctly, they received this staten^t: 
’*You answered 0 (or 1, or 2, or 3) of fibe last five questions correctly on the 

first try. " Those not reaching the criterion of four or five correct did not 

«- . » . 

receive the accompanying RF statenient. 

* ' i • * 

Analysis and Results 

Four separate analysis, of variance (ANOV) designs were a];^lied to 
the results of the different RF conditions. The four dependent vaiiables 
examined were posttest scores (PT), total learning time (T), total errors 
made in the proof section of the program (E), and errors per step per proof. 
Ability, as measured by total PMA scores from the pretest, was used as a 
Second factor in the first three designs; and errors due to proofs was the 
second treatment effect in the fourth ANOV design. Each design tested the 
null hypothesis of no difference in variance due to the two treatment or 
interaction effects. 

ANOV I .!^ference to Table 1 shows that RF conditions produced 
no significant differences among posttest scores. The five mean posttest 
scores for CRF, HRFr, LRF, KOR and SRF were 11.8, 11.8, 11.0, 11.6 
■ and 10. 7, respectively. Variance due to ability was significant (F = 7.35, 
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p < . 01). Mean high ability score = 12.4; mean low ability score = 10.4. 
Interactions were not significant at the . 05 level. 

ANOVn RF treatment differences in time on the computer were 
found to be significant (F = 3.23, p < .025). A Newman-Keuls sequential 

r ang ft test was aj^lied to the group means; Group 4, KCR, took sigmficantly 

* . • 

longer to complete the program than any of the other groups (p < . 05). 

The mean time (in minutes) for the CRF, HRF , LRF , KOR and SRF were 
276. 1, 253.6, 262. 9, 316.0 and 269.4. None of the other RF groups were 
significantly different from each other with respect to time. Ability 
difierences Vi/ere also significant ft) . 025) with high ability takii^ less time 
than low. The mean time for hi^ ability subjects v.*as 261.7 minutes, and 
for low ability 289. 5 minutes. Interactions were not found to be significant. 

ANOV III ,RF conditions produced no significant differences among 
- total errors in the proof section of the program. The mean errors were 
147.6, 134. Oi 166.6, 149.1, and 188.9, Significant variance was accounted 
for by ability level (F = 8.39, p < . 01). The mean time for high ability = 

129. 0,. and for low ability = 185. 5. Interactions were not significant. 

ANOV IV Errors per step per proof were analyzed to see if an 
increasing or decreasing trend in error rate might have devel<^>ed chie to RF. 
There were no significant differences on this variable in any of the feedback 
conditicxis. The mean errors per step per proof were 2. 17 , 2. 01, 2.44, 

2.53 and 2.91. .Variance due to different proofs was significant (F = 14.52, 
p < .001). The mean errors per step on the 11 proofs were S.67, 2.20, 
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1.58 . 2.57, 1.53, 1.53 , 2.35, 2.04, 3.14, 3.48, and 2.43, indicating that 
* 1 * 

|^ere.Vyf6r6 some proofs which were more difficult for the students. However, 

■ ' . ■ ■ . ■> ' 

there were no -significant inte^ctions between proofs and groups. 

' ^ s'* 

. i ‘‘ In addition to the analyses of variance, intercorrelation matrices 
of the following variables were computed for each RF group and for the 
combined test population: scores on the PMA. LS, PMA WG, PMA. NS, total 
PMA scores , posttest scores (PT), T, and E. 

These intercorrelations were computed for each group (N = 15) 

• and the combination of all of the groups together (N = 75). For each treatment 
group (N = 15), correlations of .50 and . 62 are significant at the .05 and .01 
levels, respectively. For the total group (N = 75), correlations of .23 and 
.29 are significant at the same levels of co.nfidence. Table 2 shows the degree 
to which these combinations of variables inter correlate. 

Discussion 

Althou^ time does not permit a full discussion of the implications 
the author has attached to the findings in this study, it is hoped that those 
interested in such a discussion will request a cqpy of the full report by mail. 

In the time remainii^, however , I should like to discuss what have been 
adjudged as some of the more pertinent results. 

Analysis of Variance . The results of ANOV II (see Table 1) show 
that there is a significant difference in the amount of time it took the different 
reinforcement groups to complete the program. The Newman-Keuls sequential 
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range test showed that the KOR group took significantly more time to 
complete the program than the other groups. Recalling that blank counters 
were displayed to KOR subjects following correct responses, rather than 



RF fee^dc as m ill other groups, it should be noted here that the amount 
of time required by the 1440 system to search and display counters is the 
same for tho^e with words as for those without. Hence , the difference in 
time performance of the KCR group cannot be attributed to any typeou^^display 
time It can be concluded, therefore, that this difference in ttme 

to complete the program for the KCR group was due. to the peouUar nature of 
that group feeaiack condition. And, in fact, KOR was the only group which 
did not receive some kind of reinforcement after a coprect response. This 
suggests that although the reinforcement conditions <Kd not produce 
significantly better learning, as measured by the posttest and error rate 
variables, the subjects in the four RF.cqnditions were more efficient in terms 

of time required to conmlete the program. This finding lends support to the 

■ * 1 

notion that Guthrie (1940) puts’ forth concerning reward. His interpretation of 
;j*eward is that it protects against unlearning rather than strengthening pribr 
behavior. This complex number program called for repeated use of mathe- 
matical definitions and field properties in order to complete, the proof secttons 
therein. Students who were not reinforced seem^ more ii^liped to fotget the 
ai^popriate use of known definitions and field properties t|i^ the students who 
were rewarded. As a result; they appeared to go throu^i longer search 



r 
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time for the information they had previously acquired %ach time they needed 
that information. Consequently, their performance time is slowed; although 
their actual comprehension does not seem to be impaired. 

Correlations , Table 2 shows that in the HRF group, LS, WG, and 
PMA negatively correlate significantly (p < . 05) with time for completion of 

“ i ► 2 ^ ‘ H 

the program. This negative correlation between ability and time suggests 



that the students who had high ability scores did in fact perfoirih more 
efficiently in terms of time to criterion. Since this is the only group in which 
this trend is found, it implies that the HRF condition tended to ^'spur the 



students on" even though this increase in speed did not seem to' influence 
learning significantly. This further validates the hypothesis that performance , 



i.e. , time to complete the CAI unit, can be enhanced by RF but thalf’feaming, 
measured by posttest scores and errors committed in the program, is not 
affected. 



The HRF group- is- also the only individual, proup which showed a 
significant negative relationship betv»een WG and 2 as well as between PMA. 
and E (p < . 05). This relationship can be accounted for with the hypothesis 
that performance of high ability, verbally-oriented students is more readily 
manipulated by verbal reinforcement than is the performance of those less 
verbally-oriented^ i 

Time ^ to complete the program and total errors were significantly 

i 

I 




correlated in the HRF and SRF groups (>6.8, p < .01. and .56, p < .05). 
Because a questicxi appears only after a correct answer to the previous 
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probiem and, hence, looping continues only as long as the .errors are made, 
it'Aight be concluded that it simply takes longer to finish the pr«^am if one 
is errors. However,' this correlation between, E ^n,d.T did not appear 

in Uie- combined group correlations nor any of the other gr^p conditions. 
Therefore, significaht ^positive correlations between T.and E for the IffiF and 
SRF groups may imply Something about the nature of the behavior process which 
takes place in a high reinforcement situation for subjects .who ^re not 
priestling well. Students who are unsure about how accurate their response 
is going to be may take significantly longer time to make an attempt at being 
correct in orddr to increase their probability of being reinforced. This would 
seem logical since the feedback in the other three groups (KOB. IBF, and CEF) 
was not designed to be as reinforcing as the feedback in the HRF or SRF 

I* 

conditions. 



over 



Conclusion 

1 . 

• ' •: . e 

- . • ' 1 . 

The resuRs of this study point to the significant influence of RF 
simple ,KOR in CAI learning. The findings demonstiate that the use of 
reinforcing statements following correct answers is a signiificaht factor m 
reducing the time students spend on the computer to learn a given set ot 
materials. This reduction in time, however, does not seem to be a detriment 

ing. Since computer scheduling and costs are a sigiiificabt consideration 
in an ongoing CAI operation, more research in this area is Justified to 



,tp learning 

! -'rtr-.y, . .*r 






substantiate this Ukely relationship of RF to learning time with othct learning 
materials and types of programs. 
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TABLE 1 



ANALYSIS OF VARIANCE RESULTS 
(Mean Differences Between Groups, Ability and Proofs) 

ANOVI 



rroup 

viean 



CRF 


HRF 


LRF 


! KOR 


SRF 


F ratio 


11.85 


11.85 


11.0 


1 11.57 


10.71 


< 1 



Ability 


High PMA ■ 


Low PMA 


■ F ratio 


Mean 


12.4 


10.4 


7.35 p<.01 



ANCV II 



Group 


CRF 


HRF 


LRF 


KOR 


SRF 


F ratio 


Mean 


276.1 


253.6 


262.9 


316.0* 


269.4 


3.23 p<. 025 



\bility 

Mean 


High PMA 
261.7 


Low P^'IA 
289.4 


F ratio 

5.36 p< .025 



ANOV in 



Group 


CRF 


w mm ^ m 

1 .HRF-:. 


LRF 


KOR 


1 SRF j 


F ratio 


Mean 


147.8 


! 134.0 


1G6.6 


149.1 


i 188.91 


<• 1 ■ 



kbility 


High PMA 


Low PMA 

' i * K * - 


F ratio 


EMean 


. .129.0 


186.0 


8.39 p^<.01 



F 



roup 

ean 



ANOV IV 

Dependent Variable: Errors per step per proof 



CRF 

2.7 



HRF i LRF 
2.01 1 2.44 



KCR j SRF j F ratio 
2.53 I 2.91 1 •< 1 



Proof 


1 . 


2 1 3 j 4 ] '5 


6 


7 1 8 


9 


10 


11 


Mean 


3.67 


22 1 1.58 i 2.57 11. 53 


1.52 


2.35 I 2.04 


3.14 


3.48 


2.43 



F ratio 

14.52 p<.00l 



*A Newman-Keuls sequential range test showed that KOR differed significantly 
(p . 05) from the other groups which did not differ from each other. 
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TABLE 2 



INTERCCRREIATION ^TRICES 



Group I - Common Reinforcement (CRF) 
15 



LS 

WG 

NS 

pm 

PT 

T 

E 



LS WG NS pm PT T E 



1.0 


.11 


.26 


.70** 


o 

1 




.47 


^.06 




1.0 


.17 


.63** 


.01 


-.10 


[-.4V 






1.0 


.70** 


.39 


-. 02 










1.0 


.16 


.18 


-.50* 








• 


1.0 


.24 


-. 66** 












1.0 


-.19 




1 




' 


- 




1.0 



Group n - High Reinforcement (HRF) 
N= 15 





TmS 


WG 


NS 


pm 


PT 


T 


E 


LS 


1-0 


.12 


.41 


.71** 


• 

o 

o 


-;50 


-.27 


WG 




1.0 


-.21 


.66** 


.49 


-.54* 


-.61 


NS 






1.0 


.49 


-.07 


.06 


CO 

o 

• 

1 


pm 

PT 








1-0 


o 

CO 

• 


.57* 


-.58* 










1.0 


00 

o 

t 

1 


-.63** 


JL JL ^ _ 

- A 

T 








1 




1.0 


.68** 


^ 1 
E 






i i 






1.0 



Group HI - Low Reinforcement (LR^) 
N = 15... 



LS 

WG 

NS 

''MA 

T 

E 



LS WG NS pm PT T r. 



1.0 


.57* 


.23 


.74** 


.03 


.06 


.24 




T.Q 


.41 


.88** 


.12 


.05 


.25 






1.0 


.70** 


.31 


.09 


-.08 


' 






1.0 


.20 


.08 


.18 










1.0 


.14 


-:72** 












1.0 


>-.03 














1.0 _ 



Group IV - Knowledge of Results (KOR) 
N = 15 





T.S 


WG 


NS ■ 


:.PMA- 


PT 


T 


E 


LS 


1.0 


.29 


.57* 


.93** 


CO 

• 


.10 


-.37 


WG 




1.0 


.17 


:-52* 


.26 


-.49 


1 

• 

o 

00 


NS 






1.0 


.75** 


-. 06 


-.11 


-.51* 


PMA 








1.0 


.29 


-.10 


-.44 


PT 










1.0 


.09 


o 

• 

1 


T 












1.0 


00 

o 

• 

1 


JL 

E 




_J JJ 








1.0 



Group V - State Reinforcement (SRF) 
N= 15 





LS 


WG 


NS 


pm 


PT 


T 


E 


LS 


1,0 


,08 


.54^1 


.75** 


.28 


.15 


-l22 


WG 




1.0 


o 

I 


.69** 


' i32 


-.29 


-.13 


NS 






1.0 




,00 


.14 


-.22 


pm 








1.0 


.35 


-. 05 


-.26 


PT 












-.45i 


-.30 


T 












1.0 


-. 56* 


E 










- 




1.0 



Total -Groups 
N = 75 ’ 





T»S 


WG 


NS 


pm 


PT 


T 


E_ 


LS 


1.0 


.19 


.37** 


78** 


.16 


.07 


-•i 


WG 




1.0 


.05 


.66** 


.26* 


o 

• 

1 


-.2 


NS 






1.0 


.61** 


.10 


.02 


-.2 


pm 








1-0 


.26* 


-.06 


-.3 


PT 










1-0 


.02 


-.5 


T 












1.0 




E 














lA 



Note: * p < . 05 ** p < . 01. 

LS- Primary Menial AblUties Letter Series; WG: PMA Word Grouping: NS: PMA Number 
eries; PMA: Primary Mental Abilities f ofal; Pt: Posttest Score; T: Time to complete Complex 
Numbers Unit; E: Total Errors committed in program. 
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:(»lPUrER AUGiiEOTED PLACEtCNT SERVICE 



William L. Maloy 
James R. Swanson 

Florida State Ifaiversity 

. - The Problem . The «odem college placement office 1. charged with 

. threefoW responsibility; provision of services to its registrants, 
services to ewployers. and infornation services to the parent institution. 

The *ility to discharge these responsibilities is becoaiing increasingly 
difficult, even in relatively ssiall institutions. The burgeoning nu*er of 
e-ployers seeking talent, and the heightened awareness of the Services 
a,ail*le to an ever growing nurf,er of graduates, are seriously taxing 

budget and personnel resources. 

At the s»e tine, there is a continuing natlona asphasis the 
conservation and developnent of human resources. This draws attention to 
a dimension of job selection which places increas_ed emphasis on employer- 
es^loyee matching to provide work environments conducive to productivity and 
success. Further, this em^asis suggests the long-term involvement of 

placement officials in career development. 

. As the number of registrants and employers grows, the placement 

Officer is required to devote more time to routine administrative matters 
«d counseling with students is moved further into the hm;kground. When 
the pl«;e»ent officer loses contact with registrants, his ability to 
provide follow-up services is seriously decreased. Similarly, administrative 
and clerical burdens limit time available for consultation with potentral 
employers to bring about their active Involvement with the placement office. 

The most desirable situation is one in which the qualifications, 
aspirations and specific needs, of both employer «d registrant, are 
in.edi.tely «vail->le to interested parties without a time-cons»i«g 
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file »e«ch. Furthennore. there imist be efforts nade to netch . registrant 
to positions on the basis of criteria rfiich would be indicative of his 
success in the positions. These ate most reasonable tasks for electronic 

data processing. 

fnt the study . It is somewhat surprising to note the 
limited utiliratioi of automated techniques by university placement offices. 
This is particularly true when one notes how well the activities of any 
placement organization lend themselves to machine accounting routines. The 
literature in this area, as we have -seen, is scanty. Still, it spells 
out explicitly and ia^Ucitly the potential of such undertakings. Yet, 
only a handful of placement offices have embarked upon various kinds of 
machine accounting, and even fewer have given any thou^t to computer 
involvement. Of the latter. Arizona State, Illinois, aid Wisconsin, joined 
now by Florida State University and Worth Carolina State University, seem 

to hflve generated the most inclusive thrust- 

Perhaps this condition is related to limited financas, time and 

the talent which must be diverted to accomplish such a task. Too, the 
long range availidiility of technically skilled people and the organizationa 
preciseness required tends to slow the progress of even the most aikitious 
placement executive and his administrative hierarchy. More likely, it is 
a combination of these factors, coupled with a group of worthy options -^diich 
lessen determination to accomplish this task. Endless controversy on 
centralization versus decentralization of placement services retards devel- 
opment and expansion efforts. In addition, it tends to undermine Imaginative 
organization and procedural patterns which, though initially expensive, hold 
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promise for systematic interaction between employers and employees. De- 
liberation on fee versos no fee and departmental or divisional user financing 
versus central administration financing keep optimal fiscal support waiting 
in the wings. Long range coranitments versus a "one-shot" job contact 
influence the personnel and economic posture of the Flaceront Office. Of 
course, the latter approach makhs it impossible to develop service potential 
as a data bank and information retrieval center for curricular analysis and 
human factors research. ' 

Furthermore, there is an overriding uncertainty * as - ,to whether 
universities, faced with a variety of iftstitutfLx)nalveotofaitceh£s«lihd numerous 
forces competing for financial support, should even be in the placement busi- 
ness. Some argue the functions could best be performed by state employment 
services and/or private agencies. Surely, this honest difference in philosophy 
has tended to undermine dramatic action on the part of placement officers who 
might otherwise have conmitted funds to prove the uses of these more sophisti- 
cated tools. Perhaps they might even have blazed some trails of career 
planning and of human resource engineering. 

While it is true that public and private agencies have made a most 
promising start in the use of modem technology fpr placement functions, there 
is some indication that vast nuxnbers of professionals T^ill shy away from these 
activities because they are fee-charging and because they have some of the 
sane. historical stigma that surrounds commercial employment agencies. In 
addition, heretofore mentioned philosophical and practical obj^tions plague 
the tJnited States Employment Service, and it appears that they are of 
sufficient magnitude to make -it unlikely that this agency will advance in 
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computerize J placement services for professional school graduates in the 
foreseeable future. Furthermore, these groups cah* give little thought to 
information needs and to potential data banlc research at the institutional 

level . 

Thus, it seems to the inve’stigator^'that professional organizations 
such as CPC, ASCUS, and NEA will do much to advance the State of the art and 
the development of systems compatibility. The employment service will fill 
the void, in a traditional sense, in those instances where univerisities 
neglect or abffldon their placement function. Yer, there remain imaginative 
new roads for placement offices which are adequately financed and philosophically 
coianitted to the exploration and develo^ent of such programs. 

The Matching Concept . The concept of the matching progrmn is direct 
in approach and practical in design, basically, it is a rapid file search to 
locate individuals or positions, based' on quantifiable parameters which relate 
to a success environment. 

The matching process is a simplification' of the mentql processes used 
in manual matching of persons to positions in placement activities. When a 
manual search is initiated, placement personnel keep in mind certain gross 
boundaries which serve to eliminate or to include persons or positrons 
depending upon which is sought. The present system establishes these boundaries 
on the basis of information supplied, and it systematically eliminates or 

# ^ m 

includes file records . 

The purpose of utilizing machine processing for this task is to let 
the machine do the "papcr-shuffliiig" (which it can’ do rapidly and efficiently), 
thus releasing staff personnel to do essential career x:ounseling and administrative 
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Vork, The load of paperwork ^ich attends the functioning of a placement 
center is tremendous; by pXac.inS organization and the filing of certain 
pp^itions on established machine procedures, it becomes mostly a clerical 
operation which may be handled by less-skilled personnel. 

Ihilosophical Framework . The system developed for this pilot study 
may be ithou^t of as "inclusive." in that it makes a concerted effort to get a 
maximum ruunpiber of. .potential employees and employers into contact with each 
other. While .it does. accept or .reject registrants on certification factors, 

: geograj^iC limitations, teaching level categories citizenship requirements, 

V „ . . ^ 1 - £3 fext that :the?e. factors are essentially set by overriding professional 
considerations, .system acceptance of several personal choices, previous 
■ training, and stafe or local, regulations. Varieties of other items are 

included for priority racing purposes and are intended to help interested 
parties- proceed through a systematic screening process. Thus, we have hoped 
to obtain tn avimiim contact, within discrete limitations introduced into the 
■ - .. job. description and/or the employee* s desires. For example, we assume the 
integrity of an employment official’s request for a senior high school 
(10 - 12) biology teacher.. in. a,. specific community. Our responsibility becomes 
pneof finding as many persons appropriately trained for this level and willing 
; to accept the geographic area on a- first, second, or third choice basis, English 
teachers (even those who might be mlling to give biology a whirl) and junior 
high general science persons will, indeed, be excluded. 

yhen computer "matches" have been obtained on these minimal 
r requirements, other important ingredients (special curricular teaching, 
f extra-curricular responsibilities, cultural environment, and the like) become 
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the prerogative of employment officials and the potential employee for 
consideration and negotiation » - 

The system is presently limits, to professional education employment 
opportunities. This v^s done, to facilitate the. development and control of 
matching vafiabler; the availability, of a large pilot group and a wide var^.ety -4 
of reported employibetit opportunities • - At. the same time, the study was done 
with an eye toward easily prepared auxiliary programs for all placement 
office registrantiL 

A wwV'bhduld be said ^out the fiscal philosophy which underlines 

.. ■ ■ ■ ■■■ . 

the study. ‘Xti the be^"-sensey. it is not- intended to be a self-supporting, 

^ • * * 

Job-seeking niachlne service. Persons - could readily be charged for entry into 

■ ' ' 'l> ill.*: 

the system. Indeed,, .findingbour students. j.obs with suitable employers through 
orderly and sophisticated procedure.s is an inn^i^ate, legitimate goal of the 

system. Nevertheless, tli'is^. alone, is below the expectation of the investigators 

■ ? -Ir-- 

and the potential of "thd='€6ols involved. To us, the large university placement 
office of the future will^ betoke a repository for, data on the institution's 

• " JJ • * * * 

graduates for location imd’IRji^ow^up studies, research in employment patterns, 
and evaluation. In this li^CC;*' -i;t beooiK^ a unique ccmponent of research 
activities attacking the elusive^ variables of career choice, career planning, 

v» — • > • - - 

and the opti^l utilization of hukan’ resource* .Allo^nng this possibility, 
one becomes less enamored with a fee-f or- the- service approach to financing. 

Quite to the contrary, one begins to explore me^s of bringing all of our 

• • • . ** * 

graduates into the system ahd ruintai’ning contact ^th them over a period of 






years so that they are a readily" available.. s our c.e of i^ormation for continued 
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analysis and evaluation. 

* '• * . ■ " , t , ^ • • 

• - . • • • 

It cannot be diiplias'ized too stronglyt. that the .i^achine process is intended 






.i', 



to augment rather thiiii replace established placement procedures. The human 
relations aspect of placement is tremendously important. Proper career 
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counseling involves a face-to-face relationship Which cennoj^ be assumed, by 
machines no master sc^M-s.tic at^ . the prpgr aqiming .. . . |lp thing about the 
system described: here is intended. to be "final.";, rather,, the output from the 
progrA :shoul4 .be considered only as. .".su^estions.";. >diich. the computer makes cn 
the basis>iof:.ayailable and admittedly incomplete -- infomation. 

The-. Job -i Seek Person. Procedure. This procedure is followed Whenever 
a job has been listed, and an individual is sought who can reasonably be ex- 
pec ted to fill the position. The .agency, listing the position .with the placenient 

^ ■ •- i”. ■ -.i-. . •■•jCT. ‘.r.- . •• 

office establi8hes-the.bc^daries..tjtirQi^h .the information supplied on the 
vacancy listing forn«.-';-This. data .may be operationally divided into two categories: 
matchvparmneteps ' and information parameters . 

Xnfprnatipn-parsDeters. are . itei^ pf information which are. supplied to 
potential candidates for. their own -use in judging the position Which is reported 

.. . : nl'i; V-. .I 

to them, i These do not esjtablish boundaries for the machine, but they may cause 
the candidate- to reject the position, or. to place it in a low category. How- 
eve.t, .-pj^y .may- alf| 9 ;.haye.^tbp, .opposite effect — to place it in a higher 
priority^j.. In. any..eveiif;. .|3j®y 4° machine match but simply appear 

as itemSi^on . tb,e .hard qopy ^rec.eiyed by the, jc^didate. Examples of items which 

^ . • • - ■ ^ .-A.. 

appear -as only .i^^ri^^.io^ are "Rural School," ."BSCS Classes," "Team Teachlr^^," 

• tt'W m » ■ m m ^ • _ ♦ 

* ' ■s'r' ■ ■ '• rv'. ••■ ■-. . . . .. . . . .. 

. .Match -parameters are chosen broadly to conform to areas within the 

-.-Z-r.xxr •''ioj:.;;- . -v . .. . . 

concept of ./.'success enviconment" and to establish those boundaries around the 

J '-.fS.: : :-r;: - :.r. . *.. -jri, . . ;J. ■- J 

position .which would, .exclude- persons not qualified for, or not desiring positions 
which are thus described. The definition of a success environment (in which 
employer: and employ<se. arc. hqppy .with .each other) must include these parameters. 

Other, parameters, .noj^- directly related to the success environment, are also 

17. . j.,. *7 j-]:. ' .. \zi' i 'i I’ni.r. ,.r u:- 
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used in establishing these boundaries. The employer m^ specxfy a requirement 
for a master's degree, uhich «ould eliminate all «*o did not hpld this degree, 
no matter how well qualified otherwise. The employer may specify, that the 
applicant be female, for example, in teaching positions in physical education, 

which would, of course, eliminate all male candidates. ■ :• 

While salary is a parameter closely associated with., the success 
environment, it was decided that this, should remain as an information item 
on this prograa. Studies undertaken by the Placement. Office indicate that 
salary is relatively low in the priority of rcasons..for accepting certain 
positions, in addition, in education at least, salary .is ir.plicit in the 

Stated geographic location preference. , . • 

The reasoning here also takes into account the., fact that, every system 

has a somedbat different salary scale. The increments, or steps along this 
scale, vqry considerably in amount as well as in thc jaethods of . attainment . 

It was consi.dered that although salary could easily ,be.useji.as,a. boundary 
factor, the range of preparation of teachers, as as, the multiplicity of 

salary schedules,, would prohibit a meaningful evalHStion; of mhere an individual 
would fit on any given scale. Therefore, it was. decided .that .4|h^ .dnimum 
salary for the ,»sition would be reported and that the individual, receiving 
the notification would decide whether or not to pursue the .position further. 

The operation of the propam is straightforward. The position is 
brought into the machine's menwry, and the person file is sepnned-for an 
individual who is (or will be) qualified in the area requir^. When such an 
individual is located, the geographic location preference of the individual 

is checked against the position location. 




168 



* The g^graphiC; location. Is a coded parweter -which consists of (1) a 
b'toiad geographic region; (2) a state; .and (3) -a region within that state. The 
bode iS’ desired to provide several degrees of generality of inclusiveness . If 
the state' code refers to Florida, the region code indicates a county or one of 
nine arbitrarily assigned regions of Florida. .Other state codes contain a 
region code ‘only if one- of. the -standard, metropolitan areas, designated by the 
Bureau of ' the '’Census, is specified. . (The two metropolitan areas of Florida are 
noi: included f) - - - 

The individual 'may sel^t three geographic, location preferences, in 
descending order bf’'prib«^ty..‘ The. program compares first the broad region 
to the candidate's first preference. If, there is a match, the program continues 

to the state pfeferehce'.' Tf therh is a match, the program next compares the 

-. -.-j- ■ - ■ 

county or metropoHtati'^area. If 'a match occurs, the program seeks the next 
match parameter. no matcli occurs on the candidate's first preference, the 
program 'continu'i^s to -'the secondh and third. If no match is found, the record 
is discarded and 'the next'- reicord in the proper certification area is sought. 

The -bandidhte may‘-be as specific in stating his preference as he 
wishes. He may’ ihdicafe'dnly one very specific preference, or he may wish to 
show three very'geherar preferences..-'! In .the former case, he would never be 
notified of a pOsitioh from 'any county or state, unless the exact location were 
entered into the syst'etn.' -Th’ the latter case, fojr.. example, if the candidate 
specified (generally) the southeastern .U.S.'A.,.,; he would be notified of all 
appropriate positrons ' C^P to an arbitrary maximum) . within the geographic area. 

The final match parameter, is .-position .typer -that is, elementary, inter- 
mediate, junior high, etc. 'T-he code^.-fOri this. .parameter is inclusive and 
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allows a great deal of genetality. For exasiple, the candidate may specify 
"elementary" in which case he would be notified of appropriate positions in 
grades 1 - 6; or he may specify grades 4 - 6, whicK would eliminate all 
primary positions for which, the candidate may be. .ptji^pfis.e, qualified. 

’ If a successful match is obtained pp.,§ll match. parameters, the 

machine prepares an output record cop*-aining all; information items which 
are utilized in preparing notifications to the persons. 

a' simple computational. procedure allows; program evaluation, of the 



candidate's geographic preference and position-, type preference,?. ^ 
comparison of priorities on the three geographic locations and the, two 
position-type preferences produces a 1-digit .c?de,s*ish ,i? .used, to assign a 
value to a particular successful match.. Afterr.all output records are prepared 
for a candidate, the priority values thus .prc^uced are used in del^emining the 
number of notificationsv.to be sent... For example, if a maximum of 10 
notifications is to be allowed-, .;the .10 records carrying the highest priority 
values will be sent, while any remaining ones-yill .be discarded., , 

The Person - Seek - Job Procedure. .,, This. procedure is the obverse of 
the Job-Seek-Person Procedure. The sane matching parameters are used to 
escAblish boundaries for inclusion. Here,, ^e person's record is entered into 
memory, and the file of available positfona is scanned for appropriate 
positions. The geographic location and type-ofrpps.itipn preferences are 
■ evaluated in the way de,scribed above, and an. output repord is written. 

The two match procedures are not. separate programs, but are integral 
parts of one program. In addition to a periodic run-by of all active candi- 
dates, provision is made for processing individual entries to the system. 
Individual candidate or position information is punched into cards, which are 
then read on the card-reader for processing. 
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• Th4 Output Progr^ » In view of the serious outpUt-bound status of 
the IBM l^ VO, if was decided that icatch progran output would be recorded on 
disk for later preparation. 

This approach allows mucH infomation to be recorded in a coded form, 
with table look-up procedures to supply the appropriate literal output . 

The- output from the Job-Seek- Person Procedure is in the form of a 
list which can either be forwarded intact to the employing agency, or edited 
and revised 'by the Placement Office staff prior to forwarding. The list 
would contain biographical information on each candidate, certification and 
experience -data, and special preparation of interests which might be of value 
to the employing agency. 

Output from the Person-Seek-Job Procedure would take the form of a 
. notification to the person, stating pertinent information about the job and 
giving the nanie and address of the contact person at the employing agency. 

- . The match progran contains provisions for an arbitrary maximum of 
notifications. ' This provision eliminates further notification to either 
persons’ or employing agencies unless feedback Ts received on action taken on 
previous notifications. 

Pile Maintenance . Although file maintenance procedures are beyond 
the' scope of this pilot project, certain provisions have been included for 
ease in later progranming. Each candidate record contains 10 positions, or 
"switches j” which can be used fOr various identifiers » In addition, provision 
Is made to record the nunbet of times each record is matched for notification. 

• .V ^ 

IJhenever a record is matched for notification, the current date is 

• .> 1 - 

entered in the record. If no feedback is received and recorded via the 
maintenance prdgrmir for two months, the record is inactivated and will not 
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be avail ^le to the match progran. Deletion cutoff dates have not been 
est^lished, but the maintenance program will provide for deletion of a 
record after an extended period of inactivity. Two types of inactivity ate 
recognized: "updated inactivity" in :diich the candidate notified the 

Placenent Office that he is not seeking a position but irtshes, to remain 
resident on the files and "regular inactivity;' in which ^^e candidates 
supplies no information. The latter will be routinely dented. 

The file maintenance progran will provide for. the addition of 
candidates and positions to the respective file on a batch processing basis. 

Progran capAilities will provide for upd^tj.^ of active md inactive 
r^ords to include current employment. Biographic^ infoimati^ and other 
"it^ of the registrant’s record also way .be u^aie^_^th this prpgran. 

The job file will also be maintained with this p|^rp. Jobs which 
have not received any references will be delated j^riodipallx. and Jobs which 
arc reported filled will be deletable through the console (although not 

^ysically removed from the file). - 

The physical deletion of both candidates and positions will occur 

when additions arc merged into a new disk file on a perK^ic hasi*- 

, ^ f , * " * 

other words a disk-to-disk transfer, with margin of all records in the add 
area and deletion of all tagged records, will be perfoTO^ M n^essary. 

Conclusions . From the system’s logic, from the progras developments 
and from the computer demonstrations, it may be concluded that the Florida 
State University Placement Office can successfully incorporate electronic tools 
into the registrant-employer assistance function. These tools can improve 
and extend the capabilities of manual search and, at the same time, free the 
pl*:ement officials to perform their professional functions of iuterpretations, 
•miration, planning, career counseling, coordination, and so forth. 
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As to the. .reporting and analysis function for our institution, 
the placenenh officer will have access to an orderly system, including 
retrieva characteristics in keeping with existing reporting demands. In 
additi^, the system is "open-ended" to allow for immediate and continuous 
reevauation of information storage, reporting capabilities, follow-up 

techniques, and employment anaysis. 

While economic considerations might be important to- some institutions 

considering the Implementation of such a system, it appears to the investigators 
that, econoidcally. it is not feasible to consider anythin* less at Florida 
State' University. Clerical support foi the iystem (coding and key punching) 
can be accomplished within the projected secretarial workstudy manpower 
available to the Placement Office. Technical assistance .and computer time 
should be budgeted initially by the central university administration with 
the strong prospect that research activities and cooperative projects with 
business, industry, and educational agencies will provide for these services 

at minimal costs to this institution. 

The paraneters for mechanized matching are appropriate to the pxlot 

study. Their expansion to include ill segments of placement activities is 

well within the system’s capabilities. 

. ' * Data collection forms have been revised in keeping with the matching 

and are, at the sase time, compatible with procedures for a manual 
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STATS AiKIETY iiXD TASK DIFFICULTY US IMG CAI I-EDIA 

Harold F. 0*Neil, Jr., Chcrlcs D. Spielberger 
and Duncan Hansen 

Florida State University 



Studies concerning the effects of anxiety on learning have, 
in the past, originated either from artificial lahorato^ 
situations or reialistic but poorly controlled naturalistic 
-settings. The Laboratory tasks "which have -been used-ii^lude 
p ^l red^ associate learning and concept forniation studies idiile the 
natural-sctting experiments have concentrated on auch.-things 
as the academic performance of students* .Computer-Assisted 
Instruction (CAI ) systems provide a learning setting in \^ich 
it is possible to obtain a detailed analysis of .the learning 
process "with materials that*, are relevant to the re^-lif e needs 
of the subject. The CAI setting, would appear to be, especially 
appropriate for investigating the effects of anxiety- upon. the 
learning process in a more systematic manner than has been 

possible to date. ; ;= ■ 

In studying the .effects- of- anxiety on learning ^7 there is 
ambiguity in the concept of anxiety that must first be^ clarified. 
Spielberger* s (1966) State-Tra^ A^iety theory distin^ishes 
between- anxiety as a transitory .stete and a relatively permanent 
personality trait. State Anxiety .(ArState) is char^terized by 
feelings of apprehension and heightened autonomic nervous system 
jittivity that vax^ies in intensity ^d fluctuates o"ver time. Trait 
Anxiety (A-Trait) refers to individual differences in anxiety 
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proneness, that is, differential tendencies among individuals to 
respond with A-State to situations that are perceived as threatening. 
This anxiety proneness or trait predisposes an individual to see 
certain types c£ situations as dangerous, particularly situations 
that Involve a threat to the individual’s self-esteem. 

to study the effects of anxiety on learning, a theory of 
learning is- needed which specifies the complex relationship 
be^en anxiety md learning. According to the drive-theory 
proposed by Spence (1958) and Taylor (1956), the performance of 
high anxious ■ students would be inferior to that of low anxious 
students on complex or difficult tasks in \diich competing responses 
were stronger, and superior to low anxious subjects on tasks in 
which the cortect responses were dominant relative to incorrect 
responses. Support for this relationship is reported by Spence 
(1964) in eyelid conditioning by Spielberger and Smith (1966) 
using serial learning, Wei tamer (1963), using paired-associate 
learning, and by Denny (1966) using a concept formation task. 

The present study will invest;igate the effects of anxiety 
on performance on a CAL learning program that is divided into 
difficult and easy sections. 

' It was hypothesized that subjects would respond to difficult 

CAI materials with increases in self-report and 
Idiysiologic^ measures of A-State. In other words, it was 
expected that A-State measures would increase during the difficult 
of the program and decrease during the easy section. With 
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respect to the effects of A-State upon performance, it was 
hypothesized that subjects who were high in A“State would make 
more errors than subjects who were low in A-State on the difficult 
task, and that the relationship would be reversed on the easy task. 
The State-Trait Anxiety Inventory (STAX) (Spielberger, Gorsuch, & 
Lushene, 1967) was used to assess A-Trait and the phenomenological 
aspects of A-State; systolic blood pressure was employed to measure 
the physiological aspects of A-State. 

Method 

Subjects . The subjects were sixteen male and thirteen female 
undergraduates who were enrolled in the summer session of the 
introductory psychology course at Florida State University. In 
order to satisfy a balanced designed criterion of the Biomedical 
Coiq>uter Programs (Dixon, 1967), three male subjects were dropped 
from the data analysis; the two male subjects with the highest 
A-Trait scores were dropped so as to maintain equal mean scores 
for both males and females. And, finally, another male subject 
was dropped as his total errors on the easy section were five 
standard deviations from the mean. 

Apparatus and Program Description . A CAI typewriter terminal 
using an IBM 1440 System (IBM, 1965) presented the learning materials 
and recorded the-snbjects ' responses. Blood pressure was measured 
by means of a desk model Baumanometer model 300. The STAI measured 
both trait and state anxiety^ 

Psystudy, a CAI program used in this study was written in a 
linear format using COURSENRITER I (IBM, 1965), an author 
prograamer language. This program was conq>osed of two main parts: 
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the difficult section, requiring tl.^ subjects to prove the field 

-r. * 'I ^ ^ . . 

properties of complex numbers; and an easy section that consisted 

■of problems on compound fractions. 

The prograinming logic requir»^d the subject to solve one 

■ ... . -j. ■ .. 

problem correctly before he could attjwipt the next one. After 
the first five correct problems on each section, a short anxiety 
scale was presented. In the difficult section, four different 
items from the STAI were used; ii the easy section four items were 

i:- . 

used, for a total of eight items. These items were chosen on 
.^e. basis of highest validity coefficients defivediftom Hodges 
Cl9h6),.. Tw representative examples will be given to demonstrate 

V. .■. • ..•3 - , r -i.!. -; - : v ..>o; . 

_ - -'i- -- ,4 ^ , 

£li8 tliat 3. subject V3S expected to niesi 

each of the twp sections. 



master in 









TMfflciilt Materials Example: The computer would present to 

■ ' 'J- f-A-.-: . . ^7 - 

the student a series of definitions concerning the field properties 
of complex nunters and then step 1 of Figure 1. As can be seen 



Insert Figure 1 about here 



in Figure 1, the problem is first presented and then, as in 
proofs of geometric theorems, a statement is presented for the 
proof (Step 2). From a sheet listing the field properties of 
conplex nund)ers with appropriate abbreviations, the subject 
selected and typed his answer. If he typed in the correct answer 
(Step 3), in this case S (substitution), the computer would 
respond "correct" and go on to the next problem. If the subject. 
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* however, responded incorrectly^, he would he given another exaaiple 
of the principle (Step 4 ) and told to try again. The subject 
would continue to respond to this item until he entered a correct 
answer. 

Easy Materials Exanple.. The computer would first type the 
material presented in Step 1 of Figure 2. The subject was then 
expected to answer correctly Step 2. If the subject typed the 



Insert Figure 2 about here 

* 



correct answer (Step 3), the computer responded "good" and proceeded 
■- to the next question. If the subject responded incorrectly. 

Step 4 was presented. The student: continued to respond until he 
emitted a correct answer. - 

Experimental Design . The experiment was divided into, four 
periods : Pre«Task Period, two. Performance Periods (Difficult and 

Easy) and Bost-Xask Period. -The procedures Xbr 'these periods 
were the same for all subjects. Durlng/the' Perfotiiance Periods, 

'~z ' . 

'the subject first progressed through the difficult learning 
-‘j'auiterials, and thc» the easy, learning materials. The A-State 
measures, SBP and STi^ A-State, were taken at the end of each 
period. Measurement of SEP. always preceded the administration 
of the STAI except during the Pre-Task Period when SBP followed 

i« 

the STAI. In additlop^ a short anxiety scale was Inserted in the 
two Performance Periods. 




The experiment was conducted by tvo experimenters and up to 
eight subjects could b.e sane .time. The design of the 

experiment minimized -the possibility of any systematic influence 
due to experimenter since all anxiety measures Here taken blind 
and no experimenter took a complete s^ieso^ ai^iety measures 
on any single student. 

: Procedure . Subjects were seated at- a CAI terminal, each of 
idiich was located in--a sound-deadened,- air-conditioned room. The 
subject remained in the room ''for* th‘e -entire experiment. The 
experimenter entered the experimental room only to read instructions, 
.'adUnl^S ter* -the. anxiety s' ales,* and assess blood' pr^sure. 
c:?:-: - :Pre-Task- Period . Each*-'bub‘ject was* asked: to read an introductory 
bcK^et-- that contained instructions for Inducing inild stress as 
well as directions for operating the. terminal. .The stress 
instructions were: >:lx .£. 

has been found-?±h^. 'success in thia program does 
not require mathematical or quantitative ability. . . it 
-r -^requires ins tead^-l thei* ability to 'make the same kinds of 
abstractions and generalizations that you are expected 
to. make in many- college- courses'. 

After ^reading the initruciions, the ^ called c . experimenter by 

meansi of- a buzzer. Then the STAI A-Trait 'scale was administered 

with instruction to indicate how "you generally feel." Following 

this s'cale', the subject was given the STAI A- St ate scale 

with -the instructions to indicate how "you feel right now, 

that is,- at this moment. '*^'*-^ Each of these scales required the 

subject to respond in a STAX test booklet. To empuasize the 
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differences between instructions, an experimenter read them aloud 
to the subject. Upon completion of the STAI scales, the subjects* 
blood pressure was taken, four to eight times until it had been 
st^ilized (two readings in a row, not differing byiMre than 
2 mm. of mercury). After each reading, the blood/ pressure cuff 
was removed so £s could operate the terminal. 

Difficult Performance Period. The subjects then started the 
difficult section of Psystudy. After he had responded correctly 
five times, a four item anxiety series was presented by the 
computer (How did you feel when you were working on the complex 
numbers?). \Jhen the subject had completed this anxiety scale, 

« . . • *jj* • 

he continued through the remaining items of the difficult section. 

At the end of this section, SBP was taken once by an e'xperimenter 
and the STAI was administered under retrospective state instructions. 
A»u Experimenter read the following state instructions: 

Fine. Now I want you to fill out this questionnaire 
again. This time I want you to circle the appropriate 
number to indicate how you felt while you were working on 
the complex number program. Read each it^ carefully and 
check it according to how you felt (this word was mnphasized 
by an experimenter) while working on this program. 

The subj-'.ct responded in the STAI test booklet. 

Easy Performance Period . The easy section of Psystudy was then 

begun by each subject. A four item anxiety scale was again presented 

by the computer after the subject had responde'^ correctly 'five times. 

These four items differed from those items presented in the 

Difficult Performance Period. The subject continued the program 

until the easy section was finished. At this point, an experimenter 




tddk thd'sttbject*'^ SBt once, and the STAX wa« administered again 
unddr' retrospective state instructions. ;An experimenter read 
the' following instructions: 



•‘ Good’. ' ilow I want you to circle the appropriate ■ 
ninuiber to indicate how you felt working through 

■ the edmpound- fraction 'program, Rememher, respond to .. 
each item according to how yo”. felt while you were working 
on the fraction program." ' 



The sub ject again xeL-ponJedliii\-tbn.5TAT^:test booklet. 

' ■* Post -Task Por5.od . Fbl.’. 0 'w.-i>g- tli'i- aanii-Ki-s tratittorof- the STAI 

in the E^sy Performance 'Failed subject 'was asked to- wait 
three minutes' 'at the tcrm?-nal bvi told that after a brief 
period, his b'l'Ood pressure x^ould’ again -be taken. SBP was- 
measured 'and the STAI ‘.::as given under standard state instriictxons 
''after this three minute waiting period. The subject wa.- then 
'a^ked* fdr^is biographical data, including number of math- courses 
he had t^en previously. - After this information was collected, 

each subject was debriefed . • ■ 

' '• ' ■ Pesu lts ■■ 

• 

The dependent variables in thf.'s Study were STAX A-State 

Scores, SBP measures and cr.:oi.s on the Difficult and Easy CAI 

tasks The effects of experimental conditions on each of these 

•b : ■ . • - :/i : - , V . . . 

measures was evaluated in an analysis of variance design, in which 

Task Periods and Sex were the liideperident variables . Errors were 



further examined as a f unctic-ii of STAX A-State scores . 

STAI A-Statc. The mean z\~State scores for all subje^'^s.* in 

I : ; ; .:r.L ^ v . . . : : . 

the Pre-Task Period, the two performance periods, and the Post- 
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TasV. Perfo'd "are presented 4-n Figure 3. It^may be noted that the 



: . ’ ! : . Insert Figure 3 aboift heare 



“■ tha STAI A-State scores increased from the Pre-^Ta^k. Period to the 
Difficult. -Task: -Period, decreased in the Easy Task Period, and showed 
no change from the Easy Task,Period to the Post -Task Period. These 
data were evaluated in a two-factor analysis of variance of design. 
In this' analysis, Tasks-Periods was the within-subjects y^iable 
and Sex was the between-subjecjts variable. Only the main effect 
of Tasks-Periods was. significant =.13.448, ^ = 3/72,p .001), 

indicating that the STAI A-State scores differed over the four 
periods and that., there was no.-difference in mean A-State scores 
for men and women. Differences between mean A-State scores for 
each period were.*;.eivaluated by t tests. These analyses indicated 
that mean A-State-' scores in the Difficult Task Period were sig” 
nificantly different from all other periods (p. < ..p5) apd t^at 






mean A-State scores in ithe Easy Period were 
different from the Pre-Task Period (p as .^ell as the , 

Difficult task Period. (p:i< .05 No other., ^naly^es were si^ific ant. 

Systolic*Blood- Pressure . .jThe •rocsn ,SBP. values obtained in the 
periods corresponding to those. fo?r which STAI. A-State measures 
available are presented' in. Figure 4.. ^.e mean SBP_ increase 



r Insert. .Figure 4 .here 






were 



iticreased during Difficult Task Period, decreased during Easy 
lask Period, and showed no change from the Easy Task Period to the 
Post“Task Period. Both main effects obtained from the two-factor 
ff....lysis of variance were significant; Sex, * 11 <44, ^ * 1/24, 
p < .01) and Tasks-Periods (F = 8.54, df - 3/72 p <.01)V These 

results indicate that SBP reliably changed over the Tasta-iPeriods 

- * 

and that, in addition, males and females differed in their SBP 
.response. Differences between nean SBP scores across periods 

z. • 

-- •• r ^ 

were evaluated by t-tests. These analyses demonstrated that SBP 
scores taken in the Difficult Task Section were significantly 
difficult froai all other conditions (p < .oS). Mo other Comparisons 

were significant. 

Errors with CAI Tasks. Both performance ^riods were divided 
into two sections. The first section of the Difficult Task Period 

: p” Ft- . - - - ^ . -b - 

-« . - . , - , I - 

consisted of the first five items (proofs), Diff/(l-5); the 
second section consisted of the remainder of ’ the items i>if f /(6-17 ) . 
Similarly, the Easy Task Period was subdivided into tiid' sections; 
Easy/a-5). and Easy/ (6-16). ' The "f oui-item anxiety sede followed 



the first sectira in each Task Period. The me® nodifeif of errors 
per item for the first ®d second sections of the Difficult ®d 
E®y T®ks are presented in Figurd -yr' Ir iiiy^n^«d 
me® errors ^r item is the greatest for the Diff/(i-5) section, 
at ® intermediate level for Diff/(6-17) ®d that errors fall to 
almost aero in Easy/(l-5> and E®y./. (6-16). .tasks. Errors were 
evaluated in a two-factor analysis, of y«riance with Sex the 



* .. - 






the within- 



betiieeii-suhjects variable and type of section 
subjects variable. Statistical significance was obtained only for 
type of section (F = 20.71, df = 3/72, p <-001) which dearastrates 
that nean errors changed across the four Performance Peridds. There 
• were no signitU nt differences in mean error scores for isen and 

voBsen» 

Ttelatioa.of A-State and Performanc e. Since there were no 
significant Sex effects in the analy:. is of the effect of experinental 
conditions on A-State (STAI) scores nor for errors on CAI tasks, 

. the data for both sexes wt.s combined in order to evaluate the 
relationship between A-State (SlAI) and errors. Separate’' analysis 
for the Difficult and Easy Tasks were run because the levels of 
A-State (STAI) on the Difficult Task were significantly higher 

JT 

than on the Easy Task. Subjects* scores on the STAI (A-State) 

- ' * 

were divided at the median into two groups. Subjects aWe the 
median Wire designated as High (Hi) A-State and those below the 



median as Low (Lo) A-State. 

The median A-State (STAI) score for the subjects in the 
Difficult Task Period was 43. Since the scores of two subjects 
fell on this median, both subjects were dropped from this 
analysis. Figure 6 shows that the Hi A-State subjects made about 



Insert Figure 6 about here 



1 1/2 tines as many errors per item in Diff/(l-5) Task than 
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"lo A-St*te subjects, but had fewer errors in Dif f /(6-17) . The 

interaction between state anxiety and Task-Period was significant 

(F = 5.08, df = 1/24, p < .05) as was the main effect of Task 

■ Periods. (F * 6.59, df = 1/22, p < .05). The interaction suggests 

Hi A-State produced a large number of errors on the Diff/(1“5) task 

and lead to f e%ier errors on Dif f / (6-17) . The Hi A-State and Lo 
• ... “ • • * 

A-State subjects were not Significantly different' ~rr ua each other 

in age, number of math courses, aptitude or mathematical ability, 

as determined by i-tests. However, the Hi A-State and Lo A-State 

did differ on Trait Anxiety (t = 3.84, df = 22, £ ^ .001). The 



mean A-Trait scores wSre 39.67 f^or Hi A-State subjects and 30.5 



- •# 



for Lo A-State subjects. 



' Similar to the previous example, subjects* scores on the 
5XAI A-Stata administered after the Easy section were divided 
(median* (32)). Since four subjects* scores fell on this 
medium, they drop^d from the analysis. The repeated measures 
design showed that neither the main effects nor their interactions 
were significant. These results indicate that there were no 
'significant changes in errors across the Easy Task nor did A-State 

effect responding in this section. 

Discussion 



It was f^iind, as hypothesized, that subjects* responded to 
difficult CAI materials with a greater increase in self-report 
and physiological measures of A-State than was the case for Easy 
materials. Both STAI A-State scores and SBP scores increased 




during a.pifficult CAI task and decreased during the Easy CAI 
task. There were no Sex effects for the STAi A-State measures 
hut .the SBP scores for males were significantly greater thin 
fettales. The most parsimonious explc nation of -his latter 
difference is that SBP is dependent upoii physical characteristics 
such as height, weight and body buile. Therefore, since nales 
are taller and heavier than female?:- it ^?as expected that males 
would respond with higher levels of SBP than females. 

The analysis of mean errors across Task Periods indicated 
that the Tasks were appropriately labeled Difficult and Easy. 

The mean nuniber of errors per item ii; the Difficult Task Period 
was 2*17, while the errors drop to almost zero in the Easy Task 
Period. Since there were so few errors in the Easy Task Period, 
the analysis of the relationship between A**State and errors 
focused on the Difficult-Task Period. There were actually two 
levels of difficulty for this Difficult-Task Period, corresponding 
to Diff/(l-5) and Diff/(6-17) . The mean number of errors for 
Diff/(l-5) was 2.76 and for Diff/(6-17) wac 1.58^ Tl-ls decrease in 



errors would indicate that subjects wsre i.-irning I ’G material. 
For Diff/(l-5), Hi A-State subjects made more errors than 
subjects who were low in A-State; however, Lo A-State subjects 
made more errors than Hi A— State subjects on Dif f / (6-17) . There 



was also a significant main effect of Tasks-Period in the Difficult 
Task Period indicating that Diff / (1-5) and Diff / (6-17) were 
significantly different. 
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The' relationship of A^State and errors is complex. For 
those aspects of the CAI taslcs in which errors yere hi^. Hi 
A— State subjects did not do as well as Lo A“State^ subjects* In 
those sections of the CAI tasks in which errors were relatively 
few. Hi A-State subjects did as well or better than Lo A-State 
subjects. 

The relationship between anxiety and learning depended 
upon whether A-State was aroused in the situation and the 
* relative strength of competing response tendencies. These 
results are consistent with. Spielberger *s (1966) extention of 
" ■ the Spehce-Taylor Drive Theory. In this State-Trait theory, 

an index of transitory drive -(A-State) is available. It is this 
transitory drive that is related to .levels of task difficulty 
and' resulting errors. : : . vi..,.-. 
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Step 2; 

Step 3 
Step~4 



DIFFICULT liMERIALS FORMAT 



-We- will show that for all pairr of elements in C, 
‘ that their sum. is also an element of C» To do 
this, .we will select two arbitrary elements Z1 = 
(.a,b.) and Z2 = (c,d) and show that Zl + Z2 is 
■ an element: of G. 

; Zl + Z2 = (a,b) + (c,d) (Type the abbreviation 
to justify this step.) 

: Answer;- Substitution (The subject types "S".) 

: Make -sure that you are using the abbreviation on 

page 10. If R = 8 and S = 7, then G -l- 7 may be 



written in place of R + S. Try again. 



FIGURE 2 
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easy; materials format 



step i: There are two ways that this computer can write 

the fractional number three-fourths. One way is 
— The other way is 3/4. In this program, 

4 

all fractional nu^ers will be /smitten like this - 
3/4. You are going to need a little practice. 

Step 2: Type two thirds.. 

Step 3: (Answer) 2/3 

Step -4?; You should have typed 2/3. Try again. 



188 



l^ienn A-Stai:c Scores 



rigure 3 

MEAH A-STATE SCORES ACROSS TASKS-PERIODS 



I 




Task - Periods 
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Mean Systolic Blood Pressure Scores 



Figure 4 

MEAN SYSTOLIC BLOOD PRESSUBE SCORES ACROSS TASKS-FERIODS 
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Mean Errors Per Response 



Figure 5 



MEAN ERRORS PER, RESPONSE 
ACROSS TASK PERIODS 
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Errors Per Response 



Figure 6 
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Comparison of feedback by verbal definition and feedback by example on two consecutive 
steps from one of the proofs, showing typical student responses for instances where at 
least two errors were made. 
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TKS ROLE OF CAI IN THE FCRMATIVS EVALUA'HON 
OF THE INTERMEDIATE SCIENCE 
CURRICULUM STUDY^ 



Dr, William R. Snyder 
Florida State University 



The Intermediate Science Curriculum Study of Florida State 
University, under the direction of Professor Ernest Burkman, was granted a 
five-year contract in May, 1966, by the U.S. Office of Education to develop a 
sequential science curriculum for the seventh, eighth, and ninth grade levels, 
and to conduct research based on this curriculum. This research is designed 
to yield information about instructton and learning that will, in turn, be 

applicable to other subjects and grade levels. 

The strategy for instruction being employed by ISCS is a umque one. 

It takes into account what teachers, administrators, and perceptive parents 
have long recognized-the fact that youngsters vary tremendously both in their 



capacity to learn and in the rate at which they are able to learn. 

ISCS is working to build a practical program of more individualized 
science instruction. Cur concept of the teachii^-leaming situation is one in 



♦Paper presented at National Council Cn Measurement In Education 
Annual Meeting, Chicago, February, 1968. 
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which much of the student’s work is conducted independently, with the teacher 
moving from individual to individual, giving ciues, asking questions, correcting 
misconceptions, and extending concepts to new situations. In this approach, the 
learning pace is set by the student, and the level of instruction is automatically 
adjusted to his ability. 

To allow for individual differences in ability, tw'o types of student 
materials are being developed. The primary sequence for each grade provides 
the basic "story line” that every student follows. "Excursions" are separately 
bound materials that provide departures from the primary sequence. There are 
two kinds of excursions. Enrichment e::tursions are designed to provide greater 
challenge for the more able student, or for the student interested in pirsuing a 
topic in greater depth. Remedial excursions provide the slower student, or 
less well-prepared student, with special background or skills needed for 
efficient progress in the main-line sequence. Through selection and judicious 
use of excursions, the ISCS teacher may design a muiti-track program 
specifically geared to Ms own students. 

Both the core sequence and the excursions are written in a semi- 
programmed style wMch permits students tc proceed independently and at their 
own pace. (Figures 1 and 2 illustrate the text forir^.l. Tiiey are reproduced 
from the seventh grade text materials. ) Note that the text serves as lab guide 
and record book, as well as a source of conceptual and factual information. 




Insert Figures 1 and 2 about here 



Eval uatk ii Tcst^n ^ 

Instructioi*al reoC^rcL v.xcl: ^x'S Lave long wished trat thej' could 
somehow pcsr Into the bisin of each inUivlcyal .s ; "-sses through a sequence 
of instruction. If this were pcosib-.e , one could pinpoint tuo exact areas of the 
sequence that cause difficulty lor specific 1:1523 oi individuals. Information of 
this sort could obviously mile.--, ti e t'-sli ci biild and rsvismg instructional 

materials easier end much more ehic:':ii; . 

Not being able to prcee dlrsclly i-J.o a student's mind, ISCS is 

gathering a record of s^idciit respoirses in a less direct fashion~by using 
computer-assisted ir-structicr-al tecliri(;ics-. As the student text materials are 
produced and sent out to he tested hy cvx r 12,0C0 students and lOQ teachers, 
they are also programmed tor entry in^o an IBM 1500 computer at the Florida 
State University Computer-Assisted Instruction Center. Sixteen seventh graders 
of different abilities are presently serving as trial students for ISCS using CAI. 
Si^ery response mode by every child to each stop of the sequence is stored by 
the computer on magnetic h-pe. Tins infonratio-i is being used to help deter- 
mine which portions of the ciassroor.i STG.xrco si ruld be revised, eliminated, 
or converted into "er.cvi'rionj. ’ 

This use of con: 'd b rtxuctnvi (CM) in the foi'mative 

evaluation of the ISCS cevo; ; .feuM’G •■■■plements Hie concurrent 

large-scale classroom trial. The C.VI ewaination pernils greater control of a 
multitude of variables that are common to ti;e science classroom, and that 



198 



inevitably bring large-scale classroom evaluation into question. Such variables 
as teacher behavior and training, different classroom environments, over- 
crowding, lack of facilities, etc. , are minimized with the CAT presentation. 

The CAI evaluation can thereby focus on providii^ objective data related to 

i ' ■ » 

conceptual, sequential, and mechanical problems inherent in the curricular 
materials. And, since complete records of each individual are obtained, a 
precise measure of how a student’s ability relates to his daily progress is 
readily available. 

The CAI Materials 

• Beckuse the ISCS seventh grade textual materials have a semi- 
programmed, self-pacing style, and because these materials contain many, 
questions for the student to answer, they have proven to be readily adaptable to 
CAT presentation. This adaptability of text to computer programming is critical 
in producing the desired match of CAI program and classroom materials. If the 
evaluation of the course via CAI is to be meaningful, we feel that it is essential 
that students see and work with the same materials, organized in the same 
order as those of the text, and have the same laboratory experiences as students 

taking the course in the ordinary ISCS classroom. 

For the most part, the CAI program is written in linear form; that is, 
the student proceeds along a single path. This matches the main-line sequence 
of the text. However, branching is employed extensively in the sequence to 
include the all important remedial and enrichment excursions. Entry to all 



o 
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remedial excursions is based on check-up frames such as the one shown in 
Figure 2. The student, in consultation with an adult proctor, decides whether 
or not to take an enrichment branch. Each excursion is programmed to provide 
an automatic reentry to the main line at the original point of departure. 

The formative evaluation of ISCS materials, using CAI, was begun in 
the fall of 1966. This corresponded to the first year of the classroom trial of 
the textual materials. The presentation of the ISCS program by computer has 
been made possible through the cooperative assistance of the staff of the Florida 
State University CAI 'Center.* This first year of CAI evaluation was an important 
learning period in regard to how one best obtains and processes feedback. Much 
of w^at we learned from this first year's experience is now having an important 
effect on our approach to the second year of CAI evaluation. 

Before describing what we have learned about the specifics of our 
approach to CAI evaluation of the ISCS materials, let me emphasize once again 
the intent of ISCS in approaching formative evaluation of curriculum in this way. 
First, vje are intensely interested in what happens to individuals as they 
proceed through the course. We think CAI provides the surest and cleanest way 
of examining individual differences in the context of a science course. Using CAI 
to look at individual records provides the kind of evaluation which cannot be 

readily obtained in the large-scale classroom approach. 

Secondly, we are attempting to provide data about how course sequence 
and construction relate to student progress. Therefore, if these data collected 
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via CAI are to be iceaningful in the formative evaluation of classroom text and 
lab materials, we are committed to presenting, by CAI, a program that matches 
the current text. Our immediate goal is not to produce the best possible 
computer program for science instruction, but rather to produce a sound three- 
year course for classroom presentation. Eventually, we plan to turn more 
attention to the development of multi-track CAI programs for the ISCS materials, 
but our immediate concern is with text revision. 

Now, let me describe some of the changes in approach that have 

• “ I . 

resulted from our 1966-67 experience. Hopefully, this will give you a fairly 
clear picture of our present operation. I shall limit this discussion to the more 
important problems we have faced, and the decision they have prompted us to 
make. 

First, we have decided that an even closer match of computer program 
and classroom text is suitable. During the first trial year, we sought to get the 
most-precise analysis of each text section by fragmenting large paragraphs into 
specific questions^ In other words, we were frequently converting declarative 
statements into interrogative ones. For example, suppose the classroom text 
was as follows:. 

"The blade of the force -measurer is bent by the weight hanging on it. " 
The computer program would have presented this statement as a question: 

"What caused the blade of the force -measurer to be bent?" 

A set of multiplf-choice responses would then be presented, or the student would 
be asked to type a free response answer. 
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In order to make sure the student knew the initial piece of information 
given in the declarative statement, he would be told by the computer whether or 
not his answer was correct; or he would be told what the correct answer should : 
be. In this way, each correct student response would be reinforced. This > 
programming technique se-med most appropriate, in that it followed Skim^^rian 
philosophy of programmed instruction. But, this method of progra.mmvni5 proved; 
to be the basis of much criticism. It was argued, and justifiably so, that we 
could not be sure what effect knowledge of results and reinforcement were havii^ 

on student learning. Therefore, we could not attribute progress to text design.. 

" * . 

only. 

Our approach this year is to preserve the exact wording of th^ text and 
to provide knowledge of results and reinforcement only when they ape fppnd m , 
the corresponding textbook sections. For example, look at the follQWing_te^ 
passage. It is reproduced word for word in the computer program. You will 
note that knowledge of results is included in the text passage. 

2-18. Whose battery do you think did the most work? . ; 

One thing is pretty obvious by now— comparing work is not easy. 

Discuss the problem with others in your group and with the class if your 
teacher calls you together. Try to come to some agreement as to whose 
battery did the most work. 

No matter how you tried, you tould not get a complete answer to the 
question. Part of your trouble can be blamed on the fact that the meaning of 
work has hot been clearly stated. We have been using the word very loosely. 

You did , ho 'ever , learn that the following might be important: 

1) How many sinkers were lifted. 

2) How far the sinkers were lifted. 

3) How fast the sinkers were lifted. 



1 } . 
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We have also decided that programming and testing first version 
material:^ with an eye to text revision is not an efficient use of .oanpower and 
machine time. The beauty of CAI for evaluation lies in its complete and 
precise record-keepingi First version text materials — at least our first 
version material.:— rdo not warrrnt such a precise evaluation. Most of the 
chains that need to be made are gross » and therefote <[uite obvious to a person 
trained in science or science education. CAI evaluation certainly identifies 
these problem spots » but unless there is an abundant supply of manpower, and 
unless computer time is ine^q)ensive , data needed for first revision can be 
obtained more efficiently through regular classroom trial. For these reasons, 
we are pr^ently evaluating only second version materials of the seventh grade. 
Our original plan to run a simultaneous CAI evaluation of the eighth grade 
materials is being delayed until a year of classroom testing and subsequent 
first revision are complete.. 

Much to our cha^in, during the first year’s evaluation, we discovered 
that computers with CAI capability generally do not come with ready-made data 
analysis programs. Moreover, the variety of computer languages and systems 
being used by CAI groups around the country varies so much that efforts to 

interchange analysis programs have been almost fruitless. The 1500 system 

■* * • 

has the l^rdware capability for data anal 3 rsis , but no well-developed software 

* • ^ 

system for implementing the analysis. The CAI Center staff at FSJ is rapidly 
developing a set of programs that serves the general needs of the many groups 
utilizing their ecpiipment. Concurrently, the ISCS staff is developing programs 
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that will meet our more. specific analysis demands, These programs take 
advantage of an. intricate and brp^d system of frame classification which we 



began using .this y€|ar. s.;, , 

In the first.year’s program, coding of frames was limited to identifi- 
cation according to chronologipal sequenpe witMn tjie program. We found that 
this limited classification did .not facilitate the inyesti^tion of the relationships 
between different science content and process themes. A more extensive coding 



was essential: if precise jq.dgn^ntis were 



to be made regarding particular 



characteristics of the materials., .. 

The developmei)tiOf,3 ten-oharaeter, alpha-numeric cede now makes it 
possible to identify each program segment ^nd question acco^to^ to its position 
in the text sequence, according to the specific science content ^d process with 
which it deals, and according to its remedial enrichment or diagnostic 
charaeterkjJtihSj: -It is now becoming possible to retrieve these data in many 



different j^Vays, depending upon 
For example, v;e may 



particular evaluative interests. 

■ ■ ■ ^ ■».» L- ;;: . ,.-j )■ 

be interested in knowing if the difficulty of the 



concept of .kinetic energy appears to be related to the difficulty with the concept 
of speed. We can use the co^ng sche;nie to single out all parj;s of the text 
related to these two concept^, and investigate the student error rate (individual 



and/or group) for these items. This potept^^l date.ba.^ and retrieval system 
make possible any number of specific iny^stiga;^ons that are usefol, not only in 
the revis!ionj!of ?haterials, but in investing which particular content sequence 
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is most logical to different kinds oi students. Computer programs afe now 

» ■ 

y^i ng written to ^nak^ possible the hill utilizatkMi of the available data. 

Because of the many demands and problems of a first jfear’s efforts we 

« t ^ 

found it impossible to make the most efficient use of the CAl evaluation in 
revision. The data analj^is provided the revision team was fairly 
conq;>lete, but the method of presenting it proved to be somewhat inappropriate . 
We TnftA* two assumptions which we later decided were in error. 

First, we assumed that the writers would have more time tiian they 
actually did at the beginning of the revision conference to examine the data in ^ 
s^me detail before beginning rewrites. Secondly, we assun^ that we should 
provide the writers with an unbiased presentation of the data, rather than our 
own interprotatlon of what the analysis suggested about the materials. In this 

way, they would be free to mak e their own judgments. 

We chose to present the various kinds of data collected on a chapter.4>y^ 
chapter basis , and to make these summaries available to each member of the 
revision team. Let me deecribe these summaries briefly. Even thon^ the 
method of presenting the analysis is being revised, these data do exemplify the 
kinds of things we feel are useful in approaching revision. 

Each chapter report included individual and group records of every 
student re^nse,'i.e. , a complete listing (including every <piestlon) of all 
correct (ca). Incorrect (wa), and all unanticipated (fui) responses. These data 
were summarized by the computer and tabulated so that qpiestions frecpiently^ 
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answered incorrectly could te easily identified. An example of a portKm of 

• If 

such a listing is shown in Table 1. 

■ . * * 

TABLE 1 




Question 
Numbei* ‘ 



Student Number 
Decreasing Ability L«vel 








Qfote that these data can be used to study individual or group progress 
through the chapter material. From the responses shown here, it is evident 
that student #3 had more correct responses than students #7 and #8 , despite his ^ 
lowCT* Ability -level. ) - 



■ Lettei* designation have the following meanings:* ca - correct 
respon se: wa - incorrect response; un - unanticipated response. 



The rfeCfeHf'stowB iia the table includes first pass responses only. That is, the 
responses are those which occur the first time each of the students approaches 




a specific frame. 
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The chapter summaries also contained a time chart (not shown) which 
allowed for an easy oompariscm of the students with respect tb their rates of 
progress through the textual sections and laboratory activities. This year, we 
will have latency data for everyprogram frame. - 

Also included in the chapter summaries were data on the percentage 
of correct answers for each student for eich chapter. An example is shown in 
Table 2. lliese data provided a rough mt^sure of the difficulty of the diapter 
for each student, and for the total student group. 



TABLE 2 





- 


Student's Percent Correct Scores 
(Students Ranked By Decreasing l.Q. ) 






8 


7 6 5 


4 


3 2 


1 . 


Chapter, 6- . . 


88.9 


81.5 86.4 t7.8 


65.4 


77.8 66^7 


61.7 


Chapter 7 


85.3 


76.0 84.0 82.7 


70.7 


68.0 60.0 


58,7 . 


Chapter 8 


85.7 


79.6 75.5 75.5 

“ ’ ! 1 . _ . 


71.4 


71.4 67.3 


59.2 



Student performance in the CAI program was also cor:*elated chapter 
by chapter, . with their ability levels and with their achievement on ISCS subject- 
matter content tests. Table 3 gives a sample of this data. 
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table 3 




Group Product Moment Correlations 



% Correct vs. Aptitude 

% Correct vs. Achievement 
Test 



Chapter 8 


Chapter 7 


Chapter 8 


.85 


.87 


.97 


.77 


s88 


.75 



t . j_ j M comoarison of the CAI 

The summaries also incmucii ^ r-®- 

inforinauon mth the ISCS text. These comparisons were designed to give the 
revision team an efficient means of determining which text sections were reUted 
to specific CAI questions. With our new coding scheme, we can identify every 
program frame according to its exact correspondent in the classroom text. 

Finally, the chapter summaries contained proctor notes on student 
discussions and otherwise unrecorded incidents related to problem areas in the 
computer program, in the ISCS text, and in the laboratory materials. These 
notes also contained the proctor's reactions to specific problems , his methods 
of assisting students in.the solution of these problems , and his subjective 
impressions of the student's progress, attitudes, and understanding. In ■ ' 

addition, all experimental data collected by the students and all student-written 
responses which had been entered in a mimeographed student text were ^ 
presented in condensed form. 



o 
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Sven thou^ many changes in the classroom text resulting from the 
revision can be traced to the outcome of ttie CAI evaluation, the need for gross 
changes in the materials, as mentioned earlier, overshadowed these more 
specific^lteratirasr- Time did not allow all the writers to hilly digest the CAI 
data. There simply was too much to examine. As a result, we have decided to 
chai^ the format of analysis presentation. It appears to us that efficient use of 
the kind of CAI data we have requires a greater condensing of the records and a 



fairly specific interpretation of the data analysis. In other words, we feel that 



the CAI data wiii be utilized much more effectively if the writer can quiduy see 



its relationship to trouble spots in the textual materials and what this relation- 
ship suggests about h^ded changes. 

Prior to tM 1968 summer revision conference , we intend to produce 

written summaries of our interpretations of what Uie analysis means with 

. » 

respect to specific revision needs. For example, we plan for our summaries to 



provide the kind of information i^ustrated in the following paragraph: 



We find that' the conqept of kinetic energy is not being applied correctly 
by .the student in Chapter 9 as indicated by the high error rate on Questions 
9-34, 9-37, and 9*^1; This appears to be related to the difficulty with the. 
concept of speed in Chapter 6, Questions 6-12, 6-15^ 6-16, etc. It may be 
possible to correct this by rewriting the first activity in Chapter 6, with 
special attention being given to the relationship of time and (^stance of the 
moving cart. It may also be necessary to write another remedial excursion 
dealing with division of decimals. 



Ificoncri^te. sug^stions of this sort are provided, we feel that the writers can be 



helped greatly in their revision task. 

- ’ • . . ; j s 



o 

ERIC 



209 



Let me quickly summarize what I have said, and project a bit into the 
future for ISCS. I have emphasized in this report our efforts to employ 
computer-assisted instruction in the formative evaluation of the ISCS science 
course. Matching computer program to the textbook provides the key to 
revision of the classroom materials. Revision of these materials in turn leads 
to revision of the computer program. The alternating stepwise improvement of 
text and computer instruction should eventually produce both a carefiilly revised 
classroom text and a refined CAI program. V/e envision such a computer 
program as possessing intricate branches designed to provide for the needs of 
different kinds of students. This is one of the ultimate outcomes of the current 
CAI formative evaluation. As has been emphasized, however, our immediate 
concern is with text revision. 

We hqpe that, by the time the ISCS project is completed in August, 
1971, we will be able to state with a gcod deal of confidence what students, who 
take either the CAI or regular ISCS seventh throu^ ninth grade self-paced 
program, can be expected to be able to do at any given point in tbe sequence. 

The size and comprehensiveness of the ISCS sample, the nature of the 
materials being developed, and the Urge bank of easily retrievable daU being 
coUected offer unique possibiUties lor much needed longitudinal research. As 
iPAntjnned earlier in this paper, research is to be a vital part of the project 
effort. Hopeftilly, such research will provide knowledge about learning and 
instructional sequencing that will be applicable to other subject fields and other 
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levels of instruction. Some studies have already been begun but, up to this 
point, much of the time of the project staff and graduate students has been 
invested ia^veloping materials and in organizing the information-collecting 
and retrieval system. As the need for this phase of the effort lessens , we 
plan to devote progressively more time to more systematic research. 
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page 7-2 Comparing jobs that are similar is fairly easy. Carry tlis a step 

farther by making the jobs even more alike. Suppose you w.inted to 
lift a ten-kilogram block to the top of a wall one meter hi^-b. Figure 
7-1 shows three Vifays of doing the same job;. 




7-3* When the boys shown in Figure 7-1 had finished their jobs , 
which boy (A., B, or C) had accomplished the most? 



7-4. Which job was the hardest to do? 

7-5- Which boy did the most work ? 

Look again at the illustration, and you will see that each boy did 
exactly the same job. Sach lifted the block to the top of die wall. 

You may think that one way of doii^ the job was easier than the 
others, but the work done was the same in all three cases. 

As part of the activities for this chapter, you will actually lift an 
object as shown in Figure 7-1. That is, you will do the; same amount 
of work in several v;ays. You will measure the variables involved 
and try to find one that remains the same no ma tt e r how the work, is: 
done. This will give you a measure of work. 




FIGURE 1 
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naee 2-6 By this time you may have gotten numbers like 1. 2 meters or 3| 

^ . fr^nHnr\R\ arft call 



seconds. Such numbers (crataimng decimals or fractions) are called 
mixed numbers. As you might guess', you will be using mixed 
numbers, particularly decisis, fr^m now on.^ Do Checkup Frame 
2-2 before going any further in this book. 



Checkup Frame 2-2 



; 1 



Solve the following: » Check your answers on page 1 

' of Excursion 1-1. 

1) 3.7x2.5= !.. 



2) 0.9 X 4.6 = 

3) 8.46-: 0.2 = 



Perhaps the work of lifting Sinkers is different from th^ work of 
dragging them. To find out, make the battery and motor drag the 
' sinkers across the table as shown ih Fi^re ^-11^ ‘Ohce more you 
shoiild be working with five of your- classmates, possibly tte same 
ones.~ Record the information from these trials in TaT)le-2-3. 



'4 



MOTOR 



•i . « * 



battery 







•'.-‘ •..s r 



Figure 2-1 
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Name of 
Group Member 


No. of 

Sinkers 

Dragged 


No. of .. 

Times 

Dragged 


Distance from 
Hook to Pulley 
(in centimeters) 


. Tutal 

Distance 

Dragged 


Total Tinie 
for Eiragging 
(in sec) 




1^"- 


6 -«y- 






— — 




2 


5 








■ “ 


3 


4 






P*- 




4 


3 










5 


2 
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1 
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JULY I, 1967 through DeceniDer 31, 1967 
1440 System- 



/ 1 * • 



iii; !:}(!/ 



I 



■C: "Project 
Number 



Project Title 



Phase 



0018 

0019 



0020 



0021 



1040 



1041 






V 

V 



0006* • Concepts -in Social .Work 

0015 pkysics Problems foi?= Homework and Review- 

.... a . 

0016 '■’^’^uhiorMttgh Curriculum- P to 

- ~^9 f • . • - 5 ' 

' ;r - 

0017 Conditioning of Word.lfe^^ing 

Concept Attainment thru Synonyms Note 4 

Individual and Group Differences in Learning Note 3 



Note 5 

P.?.cv:0'; 

" Oi' i' -Note 4 



under TwO-^^if f erent Modes of CAI 
Definition vs Example as Feedback in a CAI 

'V 

* « 

Stimulus Centered Pro'grm 

The Effects of Idiosyncratic Reinforcement 

Conditions on Learning Acquisition of a Complex 



Note 3 



Note 3 



Nv^ers Program 

' =V:ioT • ... 

CAi Applied tq.^asic Computer Prbi|rfimming' of: M iV 






* S-jCi 



-1440^5^ teat" 



Oder 



S3 - 



..I, 



1047 

1048: 



CAI •Applied to.3«aic- Computer Programming of IBM, 

1440 System irTFortran- 

General' Chemtstrj^'Solv^^ 



<. 



CAI in the, Arts for Elementary Education’ ' 



IV 

IV 



Project 

Info. 

Form 



** 

** 

** 

** 

** 

** 



** 



** 



** 
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** 
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1049 

1050 


An Experimental Study of the Effects of Linking 
Anxiety and Basic Difficulty using CAI Media, I 


Note 6 
V 


** 

★ 


1051 


Effects of Three Instructional Strategies on 
Achievement in a Remedial Arithmetic Program 


Note 3 


* 


5001 


CAI time Usage Program Sunanary by Course 


Note 2 


** 


5002 


CAI time Usage Program Simmary by Date 


Note 2 


irk 



8 

* Included in Appendix B of this repprt_ ^ 

** Included In Appendix B of previous report 



Note 1— This is an ongoing project of the center have no ^asd 



designation. 

Note 2— This is an IBM project and has np. 



• ^ 



V f- ■ 



Note 3— Thia project la eempleted and the auBPiary report la included 
in this report. 



Sote A— This project has been terminated. 

Sote 5— This project has been transferred to thd IBM 1500 CAI System.' 
Note 6-Suiimary report is 
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— V “ 
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Project No. 



Title 



Phase Project 

Inf oraetioa 
£om 



0001 



0002 



Anxiety and Task Difficulty 
using CAl Media, II 



•w* - 



.CAT. ^oject in Introductory. 



0003 



0004 



CAI Evaluation of ISCS Text 
" and' LldK:MKkerlals 

Interaction or ^xakiAer .- 
Attitude with Praise and Bl4 



IV 



III 



0005 



- 

» 4. 



Attitudes to Teaching ^Methods. 



0006 






^^Eiqp^itaental Thesis Study in 
Public Address, S^ech Department 



0007 



Kuder 
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OOMFUTER ASSISTED mSTRDCTIOM CEKIER 
FLORIDA STATE UHIVERSITY 
PROJECT =WPORMATIO« PORM 



1440 SYSTEM 



Project .MuBdbeT 10S0_ 






Dete July 1. 1^67 

Protect -Director: Harold F. O'Neil, Jr. • 

Other project personnel: Dr. Duncan N- Hansen, 

■ Dr. Charles Spielberger,- Advisor 

- 

Project Title: Anxiety and Basic Difficulty using CAl Media. 

General Mature of Project: Investigate effect of anxiety 

process . 

^ 1 Map* -I mil ate level of ditficulty within a progra®. . . 

V ttSfereSally increase state anxiety in Ss differiiMj in trait a««j^ety. 
1: S^ftratf rtat thi* differential increase should inWract vith IQ to 
produce differential performance on the same task. 

4. Produce changes in blood pressure depending upon difficulty level 

program. 



Population: Male introductory psychology students 



Procedures: 

General Design: See appendix 



Analysis: 

1. pre*post anxiety comparisons 

2. Pre-^post blood pressure comparisons 

3. Error rate of HL and IX) A ^s 

4. Ti*e to complete program 

5. Anxiety levels (state) during the program 

6. Blood pressure changes during the program 

Data to be collected and 
method of collection: 



Expected Results: Sec "Remarks" below 

Titles of CAI courses used in Project: 

2. Compfrac 



Auxiliary or Special materials to be used. 

1. State-trait Anxiety Inventory 

2. Booklet containing instructions 
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Hcouirks: 

Anxiety Scales 

1. HTA on the difficult part should bhow highest score on state 
anxiety measures. 

2. LOA on the difficult part should increase scores from pretest 

. levels l)ut be. less than HLA. 

hla on the easy part should both decrease in staft^ anxiety. 

Performance on the program * ^ ^ . 

1. It is expected that our trials, thSt HEA'swill do poorer than LOA s 

2. The exact nature of t'he'knxibty - IQ interaction will be determined 

by the amount of anxiety engendered and the difficulty level ot the 
program. ■ . 



^ 



Blood pressure - 

1. During the difficult sentence blood pressure should increase from 
pre-program level. 

2. It should decrease during the easier section from difficult .section 

3. After -.a 5 id.nute 'wait following the prdgr4m,-';it. should, return to 

- ~ ~ ; - -• . -f: i • - . 

K^nol "ifirk _ ' s r. ; - 
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r ; COHPDTER-ASSISTED IHSTIDCTIOII CENTER 

flqkha state university 

PKOJECT UNOBMATIoil FORM 
1440 SYSTEM 






Project Waibe r 1031 



Pet e _ Scpteiiber 26'.-, .-1967 

Project Director: Dr. Howard Stoker 

Other projcct:jp«!osoaael: Shaw, Pteg^aewer 

“ * * - • * *' « -*5 • » 

Project Title: Effects of Three Instructional s^ategj.^ 

in a leaedijaiiAiiiUmetic Pro^sa. 

General Nature 'ofciPtoject: Dissertation 

Objectives : . ^ . 

PDpulatloa: Junior <I«w nchlCTerffs^i^t^tte 

Leon County Schools, 60 student subjects on CAI. 



- »• -V- 



1.1 J: 



m* • - - 






^*G^ral Design: Four g«W^li, three espeti#^t^^.ftt4.on« control 

analysis: t tests of differentias between weans and proportions, 

hits to be collected and 

of collection: Preteets, tw poettents (paper nnd pencil), 

perfomance on prograi. a(^ga>tfta (awtent. .resorda)^. 



^ I .7 



* < 

r ^ V* 

*1* » « _ ’ 






rpec ted Results: 
itles of CAI courses, !?»^i;in 1^0 j^ addsu^) 

kuxiliary or Special materials to be csed:^ Hone 

*^*”^ject Starting Date: October ' Expected C^l?ti<m, Date: 

Date of First Machine Usage: Septert>ei:, .1967 . 

■ 7 \ . 



October 27 



- ir-A 






Type of Use: Entry and debug. 

Anticiapted Total Time per Student: Three hours 

Hu^er and Length of Student Sessions: Two-four, approximately 

one hour each. 



Remarks: 
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CCKPUTER-ASSISTED INSTRUCTIOH CENTER 

4 .: . ■ - ' 

FIX)R1EA STkTE UNIVEISITT 

- ’ ** 4iA?-vf • 

pilbjECT IHFOmATIOH FOttl 

1500 SYSTEM 



• t— ? ’ 



Dat e ScptwBer 21.~t^7 



Project Ih*i>er3_2^000^ 



Project Director: Harold F- 0*Heilj:-Jr. - • — , -I 

Other project personnel: Dr. Duncan Hansen, Dr. CharlM p. 

Spielberg^.,! * ” .... . 

Project Title: Anxiety and Task Difficulty Using CAI 

J -.i 

General Nature of Project: Investigate the effect of anxiety upon 

the learning process. 

Objectives: Provide rj^HcAt^rO collect latency data on 

reaponding. - i:'-'- - V . ;.-v 

Population: 

S«iple: Int|f|pductory:-Psychologyj:st^^ - 

Procedures: * - _ 

General Design: S^ Appendix. 

* -r ^ 

Analysis: - ^ " V . ' = 

(1) pre - ^st. anxiety copparisoM:.* . 

(2) Pre - post blood' pressure co«parisons 

(3) Error rates of Hi and LoA Ss 

(4> Latency data - descriptive statistics 

(5) Anxiety levels (state) dur|^ the progr^ 

(6) Blood pressure changes during the progran. 

Expected Results: See below (ReMrics) 

Titl^s-j^C^ cp^.seS;.us4pd:4%:.-Proj^t^^ - 

Auxiliary or Special materials to he u^: i 
State - Trait Anxiety Inventory 
Bo<^let containing instructions 
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Anxiety Scales 

1. HLA bn the difficult pert should show highest score on state 
anxiety »eesures . 

2. ioA'bn the difficult* pert rfio^d increese Kores frPP::preteet 
-levels but :be less then-HLA. -ju . . 

3. Wfc md BLA on the e«iy'p«rt ihoidd both deereaM in etete anxiety. 
Errors 

l/' On tte.yei7, difficult aecti^ Aooid hefee not^^errora than 
low A* s given equivalent sbility. 

- • - 1 • - 

2. On the ea^^tion p should ha»e fewer errors t^«»- 
miood Pressure' 

1 , During the difficult section, blood preaeure jhoul;d i^reese 

2. it 'ibibiid ‘decrease dncl»E 



Latencies 

1 . bipirically collected end enelysed«. 
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COMFUTER-ASSISTED INSTRUCTION Cl 
FLORIDA STATE UNIVERSITY 
PROJECT UVORMAIION FORM 



Dat e 



Project Nuebe r 0002 



Project Director: Dr. Duncan Hansen 

Other Project Personnel: Mrs. Ora Kroalioat, Miss Marjorie 

Radler, Mr. Bob Bogan— writers. Dra. Steve Edwards. Meil 
Fletcher, J. W. Melson, G. Schwarz. 









ProJ^t Title: CAI Project ..in Int^tctpry :^s^s g^sics 107) 

General Nature of Project: Development of aatc^iils for a 

- •coaqpletely antgoomo^^^p^ • group of 

students. ("Autonomous” here ^a^^J^at^t^ student does 
not attend any conventional lectiffe presentations; total 
.dnstructfpnal experience .takes .place^^t ,tiie CAI Center.) 

Objectives: One-quarter course, presented twice - Fall 

1967; Spring quarter. 1968. Winter quarter to ^ us^ £^— 

- -5- evali»tion-and'.iwrweBiettt . Objectiwqg^qre to investigate ; 
' the applicability, merits, and costs. of j^ch a course. ^ 

Population:. College.- students, -mostly freshmen, primarily not 
science majors, with little mathematics training. 

Sample: 23 volunteers from the roster of (more than /OO)^ 

registered Physics 107 students. 






Procedures: , ^ 

General Design:-^ 'IBM 1500 computer .u.sed; CRT .^tminals, 

language - OOURSEHRUER U . 






Analysis: 

])ata to be collected and 
method of collection: 






- •:> 



Expected Results: 

^j^tles of CAI courses used in Project; Pl07, physics 1, physics 2 

Auxiliary or Special materials to be used; (1) 16am P.S.S.C. 
movies (P.S.S.C. means Physical Science Study Committee) 

(2) super-Sam concept films 

(3) tape-recorded lectures 

(4) booklet of mimeographed supplementary sheets 

(5) homework assignments and textbook used by students in 
• the "conventional" sections of the class 
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COMPUTER-ASSISTED INSTRUCTION CENTER 



FLORIDA STATE UNIVERSITY 






PROJECT INFOR^IAIION;^iMW,-;.^..^ 

■ • i.*/ •-T*“ 

Jr d;- .. 15 Q. 0 ' 5 yst^ 

”* * ••*-* .** ^ • 

Dat e September, 1967 Projept Number 






0003 



Project Director: Dr. Ernest Burkinan 

Other project personnel: Dr. William Snyder, Coordinator, CAI-ISCS 

Mr. David Dasenbrock, Programmer 

Mr. Paul Flood, Programmer 



Project Title: Intermediate Science Curriculum Study 

General Nature of Project: Development of 7 -9th grade curriculum in 

science emphasizing individualized, self-paced learning. 

Objectives: 7-9th grade science for all students. 

Physics, Chemistry, Biology, Earth Science. 

Structured to independent experimentation. 



Population: 7-9th graders, ages 12-15 (all ability levels) 



Procedures: 

General Design: Linear with branching for remedial and enrichment. 



Analysis: To be determined. 



Data to be collected and 

method of collection: Taped responses to be analyzed for specific 

content and process factors in addition to general analysis of ca 
and wa and latency. 



Expected Results: Revision of textual materials. 



Titles of CAI courses used in Project: ISCS 

Auxiliary or special materials to be used: 

Supplementary record text for written responses. 

Audio visual - flip pad diagrams for lab instructions. 









7TO* 



•t ft A «i ^ . i * 



Calendar: 

Project Starting Date: .Fall, 1966 • n,,/-. 
Expected Completion Datte: Sumne|:,; 1971 

Date of First Machine Usage: October, 1966 



! **ipi.i- 



»i0 mm “ * 



Type of Use: Program entering 



iViCr.; 



Anticipated Total Time per Student: 180 hours 

- I 

Number and Length; of Student Sessions :£;! .r .hb4r UMa^r) 



/o>: 






REMARKS: 



.-i-vcr 



r -r 
^ . * 
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COMPUTER-ASSISTED INSTRUC:TI0N CENTER 

• ♦ M 

- . i * 

FLORIDA STATE UNIVEI^ITX. . 

PROJEGT^INFORMATION FQ^. 

1500 SYSTEM.. . . 

Date November 15, 1967 Project Number^ 0004 ; 

Project Director; Ted Wilson 

Other Project Personnel; Dr. T. Smith, Committee Chairman; 

W. A. Kennedy, Dr. Duncan Hansen. 

Project Title; Interaction of examiner attitude with praise 
and blame. 

General Nature of Project; Control for examiner influences in praise 
and blame studies by replacing him with a computer. 

Objectives; Demonstrate the influence of examiner attitude in 
verbal reinforcement experiments. 

Population; 100 19-year old FSU sophomores. 50 males and 50 
females from the introductory psychology course. 

Analysis; 4x4x2 ANOVA Lindquist Type III design 

Data to be collected 

and method of eollection; Response latencies, number of errors. 

Expected Results; A positive attitude will enhance the influence 
of praise; a negative attitude will enhance the influence of 

blame. 

Auxiliary or Special materials to be used; None. 

Calendar; 

Project Starting Date; 11/27/67 Expected Completion Date; March 68 

Date of First Machine Usage; 11/28/67 

Anticipated Total Time per Student; 1 hour 

Number and Length of Student Sessions; One 1-hour session 



Remarks ; 



COMPUTER ASSISTED INSTRUCTION CENTER 
FLORIDA STATE UNIVERSITY 



PROJECT INFORMATION FORM 



nate Na'v’ejibar 1, 1967 



'-■J 



Project Nuxabe r 0005 



Prolect Director: Dr. T. A. Smith-And Dr-. Duncan Hansen 

Other project .personnel: . Mr.. . John.Jtedl - gi^aduate student co-ordinator - 

Project Title: Attitudes to' Teaching Methods 

General Nature of Project: To assess attitudes of students varioiA'iy^ 

of instruction "(Intture, discussion, reading,..and-CAl reading). 
attempt will be made to determine which of the treatments aideileatulng 

the most. 

- ... 

Objectives: ^ 

in * 

Population:: - 30 : SS from the Ed. Psy. 317 class-to_ba..used-"OnaSO 0 -systera-' 
Sample: .-.ISOJa. xotal. .In- the-expeicliiient. 

Procedures: - 

General Design: .V ‘ f. 

Analysis t' Standard'~'S t a f i . sfl c a l ~~Pro c . e i hir es, ^ 

Data to be collected and . •, ■ • ■ — " 

method- ipf’-collection: ■''' 

Expected.Results: More positive attitudes- tci-camputer^a^^ed.-in^ as 

compared 1:0 the,. ^adiM^>aal teaching met^^^ 

--discussion) . ' ' ^ ^ ‘ 

T i 1 rAT-v-c purses' .us^-in J?ro:iect.i._ -Essay- 

Auxiliary~or" Special. jnaterials' to. -be. . used ; PTc 



f.-j- 



Calendar: .. ' .. ^ 

Project-Starting-Date:- Dec,., 1967’ E^ted^Completion.Date: A;px-xl..l,.. 1?&1. 

Date of First machined iUs-age: Dec. 1, 1%7 Type of Use: 

“■■ — -Anticipated Total Time--*p'er Student: 1 hour ... ' 

Number and_lsragth-of .Student Sessions: 30 students will -he run. on -the --system 



* 5 * 

(a)iL.ifiegun:|>^.€^er 1967 



Dates: 

Coding: trvr-'''=rr — " 

(b) ..Ccfliple ted. December- IO5- 1967 - 

Debugging: / (a) Begun December' 10, 1967 

(b).j- Completed 

Student~Testing^(a) — Begun ' 

(b) Completed...-; * 






Remarks.;. 
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PRO JECT..:Ilff OPTION FORM 

1500 SYSTEI4 **‘ 



Date Moveniber ,16, 1967 



Project Numbe r 0006 



ri - 



Proj^t Difs^tor: ■ -Dr: Thomas R. King, major professor 
• 1'--.- Other '^project* personnel: Mr- Ji Clark Weaver, minor- prof essor 

Mrs. Marsha Markle, author^ i-ir, Walt Blomquist, GAI editor and advisor 



Project Title: An Experimental Thesis Study as partial fulfillment 

of a. Mas ter of Arts Degree in Public Address, Speech Department. 



V. 



^ ^ n, - - 

General Nature of Project; Linear program designed: to teach 
beginning students the basic steps of speech preparation. 



Objectives: The course will be approximately 230 frames.^” The objectives 

are to develop and test the effectiveness of a CAI presentation 
of programmed learning of basic speech preparation 



Population: College students in a freshman level beginnittg . speech 

course. • —■ 

- Sample: Group I: Experimental, 25 college students enrolled 

. . . 3 : in Speechi-lOS: at FSU^^during Spring .quarter:; 1968 who 

will, receive the CAL: treatment .- - nr i . r; -.vi- 



Group II: Control, 25 students in Speech 105 who will 

not be exposed to the CCAI program. ■ _ = 



Analysis: Student response analysis, pre- and posttest$>:on the 

information included in the program, and judges* ratings of 
the speeches. 



Data to be. collected and V 

method of collection': Students* responses to ^questioning, on. the 

content-jOf Ae program and judges* ratings, of the speeches. 



— ' 



Expected Results: It is expected that those who are exposed to the 

program will score significantly higher on the content exam.— = , 
ination than those who are not exposed- to the program. The 
results of the performance examination are anticipated. : 
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Auxiliary or Special matejrjU^l ,to.be used: None. 

• •• S - mm 

Calendar: , _ ^ 

■ Project Starting DatA: ‘ 8-67 Expected Completion Date: 

Date of First Machine Usage: 3-1-68 

Anticipated Total Time Per Student: 3 to 4 hours 

Nu^er and rbength. of Student Sessions: 2 to 3 visits, 1 to 2 

* “ hoars each' ** ’ -- 

Remarks: General Design - Pretest 'o^ tjfe' inf ormat^^^^ in 

the program given to holh the experimental and control groups. 
Experimental group exposed to the program. Control group is not. 
Posttest of information in program is administered to both groups. 
In addition, both groups will be required to t^ 
examination to see if the subjects dan apply ;^at they have 
The control group will also give the speeches.^- The speeches will 
three minutes in length. The speeches will be videotaped. Judges, 
composed of instructors in the Speech Department, will view these. . 

speeches and rate them. - 
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Date August 16, 1967 



^6“j^t Nunber ' 0007 



;^p j^t>;W.r^1:^r f -Dr, Frank Benhan. , 

. r c. ptherr.Project Personnel: Dr., ^nry^ tipj^rt 

■“■*• z* ^ "*5* ♦ * “ « * * 

:?rpJ^t|,T4tl^^ .. . ■ .i.-*. . 
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2 j-: . Geiteri^;rNatu^ :p£ Pro ject : Administration -and scoring' of K^^et' 

z" -:^ef(«jB^e Record' ■ .._ ... - : J..,V * " 
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Data to be collected 

and method of collection: Collected and displayed on-line. 






Auxiliary or Special toaterials to be used: None. 



Calendar; 

Project Starting Date: 8-67 

Anticipated Total Time Per Student: 30 minutes 

Number and Length of Student Sessions: 1 session 
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APPENDIX C 



'i 



SYSTEM .UTIlfIZATION MPORT 



July 1, 1967 through December 30, 1967 

v 

14^}0 System Usage 



MV 



TOTAL TllC 
FOR: 



CAl 

reassembly 



COURSE' LISTING 



AUTOCODER PROGRAMS 



OTHEPv HOUSEKEEPING 



TOTAL SYSTEM TIME 



JULY 



AUGUST 



SEPTEMBER OCTOBER . NOVEMBER DECEMBER 



179. oi.; 7S.36r^ ; 170.72 321.^' 






3.09 



.86 






22.30.1 20.71 



27.63 



233.50 100.07 



13.34 



187.37 



.20 



349.53 



300.09 .tl69.38 



.67 



32.37 



.90 



.74* 



40.39 



343.13 211.41 
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6.02 



9.00 



7.33 
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APPENDIX C (continued) 
KOUPxS OF' i^ERMINAL USAGE 



'4-ir 



COURSE CODE 


JULY 


ADDSUB 




ARTHIS 


.13 


CAINT 




CHEM 


.30 


cbMhs;'^::?: - 


.30 ; 


cc»finJMii 


85.53 


C(»0<UMST 


21.01 


ccwsd^L 


57.65 


SDTEST 


41.57 


FORTRAN 


1.50 


FSUCW 


89.17 


FSUTEST 


6.73 


GAMEPROB 




jubp 


1-7P. 


LEARN 




LISTGEN 




LOCKHEED 


.75 


MISSOURI 




NOMADIS 


.32 


NSGLSI -i 


.43 


OHMSLAW 


.35 


PHTSDEMD 




PHJSIGS 


3;3Q 


PHYTEST 


32.92 


PROOF 


1.58 


PSYSTUDY 


56.34 


TESTREF 


.27 


T2STSTAT 


.82 


UELCOME 


102.36 


TOTAL TIME USED 


504.03 


DOWN TIME 




AVAILABLE TIME 


1078.86 


PERCENT UTILIZATION 60.08 


ElOCTOR TERM". USE 


144.19 


REMOTE USE 


89.91 


TOTAL TERMINAL 


738.13 



TDC USAGE 



AUGUST 


SEPTEMBER 


OCTOBER 


NOVEMBER 






33.09 


155.56 


44.00 


17.24 
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m ^ < 


* " % 


■ 


; 


23.78 


57.33 . 


456.71 


159.24 




1.33 


1.92 






15.90 


1.58 






-■‘M. 


7.67 


16.83 




3.42 


136.05 


73.42 




1.25 






i'c. ' . 

10.33 


* ” " 


2. bo 






1.50 


. 280,45 

3 • i * 


- 

291.12 






” t * - 


79.95 






4.08 


.25 


78.11 


281.80 


1199.61 


838.58 






85.75 


470.16 


1065.96 


1930.86 


1673.58 


27.04 


26.44 


62.13 


50.09 


49.04 


185.33 


276.06 


62.00 




14.92 


90.63 


96.55 


127.15 


482.05 


1566.30 


997.13 



DECEMBER 



45.32 



16.83 

11.17 

12.75 



2.09 

371.93 



646.05 

22.51 

1005.30 

64.26 

168.91 

29.39 



844.35 




APPENDIX C (contittued) 
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HOURS OF 1500 USAGE 






Xo1,:£;/.-v.2 . 



COURSE CODE 




JULY AUGUST SEPTEMBER 


OCTOB^ 


NOVEMBER 


DECEMBER 




- 










DAVE* 




.58 - 

a m 

■ • - -• 


C » ^ ; -N 




i --C* 




•mi . 




: 


r : 




ISCS 


Oi 


253’.8p 


“ 5. 1 « - 

• %8.34- ' 


235.88. : 

I # « • 


.101*;37 

; 4 


KFH* 


‘ : 


f V;^ 


o A-**.. 

- V . • ^ 


• -a . — • 


** 2 

• • a 

J 


KUDER 


\ 


■ k'bd"' ■ 

- 




. 1.Z5- 

- -1- ^ _ - * 


* • 5 • a 

• ‘-‘•a 


P107 


• 


2pj'.i5 




-i309i73 .: 

5. 


.146, .;44 


PSY2 


1 


- : C? 


' ; 9 8 ; i8 - 


* — — .. a 

.-13';r75.=^-c.*- 


4.09 










Ai a* 


a 


GAJES** 




21.51 




l-40i81 !• 


. 5.10 


MONITOR 




“a - “ 


■ ^11.50 

■ « 














5.00 


13.01 


ESSAY 




















7.09 


. 7.88 


THESI 












MAT 








- 


j: 9.00 


TEST 


< 
» % 


w _ 




- 


13.78 


TOTAL TIME U^D 


• 5()8.6l 


771.22 


598.38 : 


300.67 












00.00 


D014N TDC 






y • " 







AVAILABLE TIME 

PERCENT UTILIZATION OP TBC 

T01AL TIME UTILIZED 

•• rj" 

j! , * TESTING PROGRAM 

IBM DEMO PROGRAM 
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^pendix D 

DEMONSTRATION REPORT 

^« « « * m ^J****^ i 2 m ' •• 

July 1, 1967 throu^ lOecei*^ 30, 1967 
nOMMITER-ASSIStEb CENTER 



Number of 
Terminals Used 



Total Number 
Attending 



Florida State Univer^ty ■ 7 .:.- 

Dr. Ernest' Burkman, 511 Class 
Dr. Mildred Swearingen, Ed. Mminis. Class 
Dr. Marian Black, Ed. Administration Class 
Dr. William Malloy, Supervisicm. Class 
Dr. Eugene Nichols, Math EducVtibn Class 
Dr. Timothy Smith, Exper. Psycl^ology Class 
Dr. Frances Hall, Ihys. Ed. BLesearch Class 
Dr. Thomas King, Speech Clasps 
Mr. Jay Knippen, Business Mucation Class 
Miss Valerie Cha^,rlain, Class 

Dr. Raymond E. Schultz, College Teaching 
Dr. Peter Everc^t-^: Physical E4- Class 
Dr. John Champion, University President 
Dr. WilUam Quinly, Education^ Media 
Mr. Jim Swanson, Placement Bureau 
Mr. Mike Beaudoin,; Information Services 
Other faculty, staff, and students 

z i\'- 

IBM 

On site 
Remote 

University of Tennessee 
University of Ohio 

National Science Foundation Institute 
Social Studies Institute 
Chipola Junior College - Elementary and 
Secondary Teachers-,- Marianiva.,^ 

Florida Department of Public Welfare 
Florida Department of Education 
L. W. Singer Company 

University of Florida, Gainesville, Fla. 
Lillian Ruediger School, Sixth grade 
Tallahassee, Florida 

Public School Supervisors of Charlotte, N. C. 
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6 

6 

2 

4 

4 

4 

3 
10 

4 
4 

4 
1 
8 

5 
8 
8 



10 



12 

12 

6 

2 

5 

1 

10 

16 

5 



20 

20 

40 

8 

10 

11 

13 

6 

40 

15 

15 

10 

3 

18 

22 

20 

32 



20 



30 

30 

18 

7 

12 

4 

25 

73 

9 



Participating in the preceding demonstrations were representatives from 
the following organizations in addition to those already identified: 



U.S. Office of Education 

Leon County Board of Public Instruction 



Participants 



in the preceding demonstrations 



(continued) 



SACS, Paris, France 

University of Utah, Salt Lake City, Utah 
Southeastern Regional Laboratory, Tallahassee 
Florida State Department of Education 
Governor’s Educational Commission 
Clovis, New Mexico High School Faculty 
Florida State Board of Regents 

Leon County Public Schools, Instructional Services Department 
University of Aachen, Germany 
General Electric Corporation 

North Carolina State, Elizabeth City, North Carolina 
Connecticut University, Storrs, Connecticut 
Stanford Medical College, Stanford, California 



IBM, Paris, France 

Atlanta Journal, Atlanta, Georgia 

Conmission on College Hiysics •. j 

Swiss Federal Institute of Technology, Zurich, Switzerland 
French National Institute of Applied Sciences, Lyon, France 
Bristol University, Bristol, England 



Iftiiversity of Texas, Austin, Texas 
Milgo Electronics Corporation, Miami, Florida 
Brigham Young University, Provo, Utah 
University of Cincinnati, Cincinnati, ^io 
Miami Dade Junior College, Miami, Florida 
W. B. Saunders Publishing Company 
University of Georgia 

Information Sciences, Melbourne, Florida 
Carleton College, Northfield, Minnesota 



